Systems, devices, and methods for tracking and compensating for patient motion during a medical imaging scan

ABSTRACT

A motion tracking system for dynamic tracking of and compensation for motion of a patient during a magnetic resonance scan comprises a first camera positioned to view an optical marker along a first line of sight; a second camera positioned to view the optical marker along a second line of sight; and a computer system configured to analyze images generated by the first and second cameras to determine changes in position of the optical marker, and to generate tracking data for use by a magnetic resonance scanner to dynamically adjust scans to compensate for the changes in position of the optical marker, wherein the computer system is configured to dynamically adapt its image analysis to utilize images from all cameras that are currently viewing the optical marker.

INCORPORATION BY REFERENCE TO RELATED APPLICATIONS

The present application claims priority to U.S. Provisional ApplicationNo. 62/028,194, filed on Jul. 23, 2014, titled SYSTEMS, DEVICES, ANDMETHODS FOR TRACKING AND COMPENSATING FOR PATIENT MOTION DURING AMEDICAL IMAGING SCAN. The present application is related to PCTApplication No. PCT/US2014/012806, filed on Jan. 23, 2014, titledSYSTEMS, DEVICES, AND METHODS FOR TRACKING AND COMPENSATING FOR PATIENTMOTION DURING A MEDICAL IMAGING SCAN, which claims priority to U.S.patent application Ser. No. 13/831,115, titled SYSTEMS, DEVICES, ANDMETHODS FOR TRACKING MOVING TARGETS, filed on Mar. 14, 2013, whichclaims the benefit of U.S. Provisional Patent Application No.61/849,338, titled SIX DEGREES OF FREEDOM OPTICAL TRACKER, filed on Jan.24, 2013. PCT Application No. PCT/US2014/012806 also claims priority toU.S. Provisional Patent Application No. 61/834,382, titled SYSTEMS,DEVICES, AND METHODS FOR TRACKING MOVING TARGETS, filed on Jun. 12,2013. Each of the foregoing applications is hereby incorporated hereinby reference in its entirety.

BACKGROUND

The disclosure relates generally to the field of motion tracking, andmore specifically to systems, devices, and methods for tracking andcompensating for patient motion during a medical imaging scan.

There are various modalities for performing medical imaging of patients.For example, magnetic resonance imaging (MRI) is a medical imagingtechnique used in radiology to visualize internal structures of the bodyin detail. An MRI scanner is a device in which the patient or a portionof the patient's body is positioned within a powerful magnet where amagnetic field is used to align the magnetization of some atomic nuclei(usually hydrogen nuclei—protons) and radio frequency magnetic fieldsare applied to systematically alter the alignment of this magnetization.This causes the nuclei to produce a rotating magnetic field detectableby the scanner and this information is recorded to construct an image ofthe scanned region of the body. These scans typically take severalminutes (up to about 40 minutes in some scanners) and in prior artdevices any significant movement can ruin the images and require thescan to be repeated.

U.S. Pat. No. 8,121,361, issued Feb. 21, 2012, entitled Motion TrackingSystem for Real Time Adaptive Imaging and Spectroscopy, describes asystem that adaptively compensates for subject motion, and thedisclosure therein is hereby incorporated herein by reference.

Additionally, there are various radiation therapies, proton therapies,and other therapies that can be applied to patients. For example,radiation therapy can be applied to a targeted tissue region. In somesystems, radiation therapy can be dynamically applied in response topatient movements. However, in many such systems, the tracking ofpatient movements does not have a high degree of accuracy. Accordingly,the use of such systems can result in the application of radiationtherapy to non-targeted tissue regions, thereby unintentionally harminghealthy tissue while intentionally affecting diseased tissue. Theforegoing is also true for proton therapies and other therapies.

SUMMARY

The disclosure herein provides systems, devices, and methods fortracking and compensating for patient motion during a medical imagingscan, such as during a magnetic resonance imaging scan.

An accurate and reliable method of determining the dynamic position andorientation of a patient's head or other body portion during MRIscanning or therapeutic procedures is a requirement in any attempt tocompensate for subject motion during such procedures. Toward this end,disclosed herein are systems and methods that include practical opticalhead tracking capability using at least a first sensor, e.g., a firstcamera, and a second sensor, e.g., a second camera, such as a pair ofcameras, for example ordinary CCD cameras, ordinary white light or LEDillumination, and a marker or target, such as a compact, inexpensivetarget which mounts comfortably and rigidly to the subject's skeletalframe. The camera can be configured to detect any desired wavelength orrange of wavelengths of energy, including one or more of the infrared,near-infrared, visible, or ultraviolet spectra for example. Somepreferred embodiments can track head and other body motion having up toand including six degrees of freedom (sometimes referred to as 6-DOF).

For purposes of this summary, certain aspects, advantages, and novelfeatures of the invention are described herein. It is to be understoodthat not necessarily all such advantages may be achieved in accordancewith any particular embodiment of the invention. Thus, for example,those skilled in the art will recognize that the invention may beembodied or carried out in a manner that achieves one advantage or groupof advantages as taught herein without necessarily achieving otheradvantages as may be taught or suggested herein.

According to some embodiments, a motion tracking system for tracking andcompensating for motion of a patient during a medical imaging scancomprises: an optical marker configured to be attached to the patient,wherein the optical marker comprises a plurality of optically visiblelandmarks, the plurality of optically visible landmarks defining aplurality of reference points; a plurality of optical detectors, whereineach of the plurality of optical detectors is positioned to view theoptical marker along a different line of sight; a marker location filterfor determining locations of marker reference points, the markerlocation filter configured to: receive a digital image from each of theplurality of optical detectors, wherein each digital image represents aview along a different line of sight; determine, for each digital image,whether the digital image includes a view of the optical marker; and foreach digital image that includes a view of the optical marker, determinepositions of the plurality of reference points in the digital image; apatient orientation filter for estimating a three-dimensional pose ofthe patient, the patient orientation filter configured to: calculate aplurality of baseline attributes related to the plurality of referencepoints, the plurality of baseline attributes calculated based on thedetermined positions of the plurality of reference points; and estimateiteratively the three-dimensional pose of the patient, until a measureof error is within a threshold amount, the measure of error calculatedbased on the plurality of baseline attributes as compared to a pluralityof comparison attributes, the plurality of comparison attributescalculated by assuming the object is in an estimated pose; and one ormore computers configured to operate the marker location filter andpatient orientation filter, wherein the one or more computers comprisesa computer processor and an electronic storage medium.

In some embodiments, the patient orientation filter is configured torepeatedly estimate the three-dimensional pose of the patient at a rateof at least 100 Hz. In some embodiments, the plurality of baselineattributes comprises a set of principal quantities, the set of principalquantities comprising values associated with the relationships betweenthe positions of reference points. In some embodiments, determiningwhether each digital image includes the optical marker comprises:classifying each pixel in the digital image as one of two colors;grouping one or more connected regions, wherein the connected regionsare comprised of adjacent pixels with the some color; filtering out oneor more connected regions based on a size threshold; computing thecentroid of one or more connected regions; and grouping together one ormore connected regions by their computed centroids. In some embodiments,the motion tracking system further comprises a coordinate systemconverter, the coordinate system converter configured to remap acoordinate system associated with the three-dimensional pose of thepatient estimated by the patient orientation filter. In someembodiments, grouping together one or more connected regions comprises:identifying connected regions with centroids occurring within one pixelof each other; and calculating a centroid for each group of the one ormore connected regions. In some embodiments, the motion tracking systembinarizes each digital image before classifying each pixel in thedigital image as white or black. In some embodiments, calculating acentroid for each group of the one or more connected regions comprisesaveraging the centroid for each connected region in the group of one ormore connected regions. In some embodiments, the motion tracking systemfurther comprises: a tracking filter configured to generate trackingdata based on the estimated three-dimensional pose of the patient and totransmit the tracking data to a medical imaging scanner controller toenable a medical imaging scanner to dynamically adjust the medicalimaging scan to compensate for patient motion. In some embodiments, themotion tracking system further comprises: the medical imaging scanner,wherein the medical imaging scanner is configured to dynamically adjustthe medical imaging scan to compensate for patient motion based on thetracking data.

According to some embodiments, a motion tracking system for dynamictracking of and compensation for motion of a patient during a magneticresonance scan comprises: an optical marker configured to be attached toa patient being scanned; a first camera positioned to view the opticalmarker along a first line of sight; a second camera positioned to viewthe optical marker along a second line of sight; and a computer systemconfigured to analyze images generated by the first and second camerasto determine changes in position of the optical marker, and to generatetracking data for use by a magnetic resonance scanner to dynamicallyadjust scans to compensate for the changes in position of the opticalmarker, wherein the computer system is configured to dynamicallydetermine whether the first camera, the second camera, or both camerasare currently viewing the optical marker, and wherein the computersystem is configured to dynamically adapt its image analysis to utilizeimages from all cameras that are currently viewing the optical marker.

In some embodiments, the motion tracking system further comprises: oneor more additional cameras each positioned to view the optical markeralong a different line of sight, wherein the computer system is furtherconfigured to dynamically determine which of all of the cameras arecurrently viewing the optical marker. In some embodiments, the motiontracking system further comprises: a third camera positioned to view theoptical marker along a third line of sight; and a fourth camerapositioned to view the optical marker along a fourth line of sight,wherein the computer system is further configured to dynamicallydetermine which of all of the cameras are currently viewing the opticalmarker. In some embodiments, the motion tracking system furthercomprises: a third camera positioned to view the optical marker along athird line of sight; a fourth camera positioned to view the opticalmarker along a fourth line of sight; a fifth camera positioned to viewthe optical marker along a fifth line of sight; and a sixth camerapositioned to view the optical marker along a sixth line of sight,wherein the computer system is further configured to dynamicallydetermine which of all of the cameras are currently viewing the opticalmarker. In some embodiments, the motion tracking system furthercomprises: one or more additional optical markers configured to beattached to the patient being scanned, wherein the computer system isconfigured to analyze images generated by all cameras that are currentlyviewing at least one optical marker to determine changes in position ofthe optical markers, and to generate tracking data for use by themagnetic resonance scanner to dynamically adjust scans to compensate forthe changes in position of the optical markers. In some embodiments,each optical marker comprises a plurality of optically visiblelandmarks, the plurality of optically visible landmarks defining aplurality of reference points. In some embodiments, the motion trackingsystem further comprises: the magnetic resonance scanner, wherein themagnetic resonance scanner is configured to dynamically adjust scans tocompensate for the changes in position of the optical marker based onthe tracking data.

According to some embodiments, an optical marker for use in a motiontracking system comprises: an optically visible pattern comprising aplurality of reference point locators, each reference point locatorconfigured to define a single reference point, each reference pointlocator comprising alternating dark and light elliptical shapes centeredon the single reference point, wherein at least one of the plurality ofreference point locators is larger than at least another of theplurality of reference point locators, wherein the optically visiblepattern is rotationally asymmetrical about an axis normal to a plane ofthe optically visible pattern.

In some embodiments, the at least one of the plurality of referencepoint locators that is larger than at least another of the plurality ofreference point locators is centered on the optical marker in one axisof the optical marker and offset from the center of the optical markerin another axis of the optical marker. In some embodiments, the at leastone of the plurality of reference point locators that is larger than atleast another of the plurality of reference point locators is positionedwithin a boundary defined by all others of the plurality of referencepoint locators. In some embodiments, the optical marker furthercomprises: a mounting portion configured to affix the optical marker toan object being tracked. In some embodiments, the optical marker furthercomprises: a first substrate affixed to the mounting portion, the firstsubstrate comprising at least one of the plurality of reference pointlocators; and a second substrate affixed to the mounting portion, thesecond substrate comprising at least one other of the plurality ofreference point locators. In some embodiments, the elliptical shapescomprise circular shapes. In some embodiments, the elliptical shapes areconfigured to appear circular when viewed along a line of sight otherthan normal to the optically visible pattern. In some embodiments, eachreference point is no more than 0.5 inches away from another referencepoint. In some embodiments, each reference point is defined by acentroid of a reference point locator. In some embodiments, theoptically visible pattern further comprises one or more non-concentricdots.

According to some embodiments, a motion tracking system for dynamictracking of and compensation for motion of a patient during a magneticresonance scan comprises: one or more optical marker configured to beattached to a patient being scanned; one or more cameras positioned tocomprise different lines of sight; and a computer system configured toanalyze images generated by the one or more cameras to determine changesin position of the one or more optical markers, and to generate trackingdata for use by a magnetic resonance scanner to dynamically adjust scansto compensate for changes in position of a patient based on thedetermined changes in position of the one or more optical markers,wherein the computer system is configured to dynamically determine whichof the one or more cameras is currently viewing an optical marker, andwherein the computer system is configured to dynamically adapt its imageanalysis to utilize images from all cameras that are currently viewingan optical marker.

In some embodiments, each of the one or more optical markers comprises aplurality of optically visible landmarks, the plurality of opticallyvisible landmarks defining a plurality of reference points. In someembodiments, the motion tracking system further comprises: the magneticresonance scanner, wherein the magnetic resonance scanner is configuredto dynamically adjust scans to compensate for the changes in position ofthe patient based on the tracking data.

In certain embodiments, a motion compensation system for tracking andcompensating for patient motion during a medical imaging scan comprises:an optical marker comprising an optically visible pattern and a mountingportion, the mounting portion configured to affix the optical marker toan object being tracked, the optically visible pattern comprising aplurality of reference point locators, each locator configured to definea single reference point of a reference shape; a first optical detectorpositioned to digitally image the optically visible pattern along afirst line of sight, the first optical detector configured to generate afirst digital image of the optically visible pattern; a second opticaldetector positioned to digitally image the optically visible patternalong a second line of sight, the second optical detector configured togenerate a second digital image of the optically visible pattern; atracking engine configured to determine a pose of the object in sixdegrees of freedom by analyzing the first and second images; and acontroller interface configured to generate tracking information basedon the pose and to electronically transmit the tracking information to ascanner controller to enable compensation within a medical imagingscanner for object motion; wherein the tracking engine and controllerinterface comprise a computer processor and an electronic storagemedium.

In some embodiments, the mounting portion is configured to rigidly affixthe optical marker to the object being tracked. In some embodiments, themounting portion is configured to affix the optical marker to top teethof a patient. In some embodiments, the mounting portion is configured tobe custom-fitted to the top teeth of the patient. In some embodiments,the mounting portion comprises an adhesive. In some embodiments, theoptically visible pattern is located on a single unitary structure. Insome embodiments, the optically visible pattern is distributed among atleast two non-connected structures. In some embodiments, the opticalmarker is internally illuminated. In some embodiments, the plurality ofreference point locators comprise alternating dark and light ellipticalshapes. In some embodiments, the elliptical shapes are configured toappear circular when viewed along the first and second lines of sightwhen the object being tracked is in a home position. In someembodiments, the first and second lines of sight are separated by 90degrees. In some embodiments, the elliptical shapes are configured toappear circular when viewed along a line of sight separated from anormal to the optically visible pattern by 45 degrees. In someembodiments, the first and second lines of sight are separated by 30degrees. In some embodiments, the first and second lines of sight areseparated by 60 degrees. In some embodiments, the first and second linesof sight are separated by an angle of 30 to 100 degrees. In someembodiments, the first line of sight is offset angularly by a firstangle from a normal to the optically visible pattern and the second lineof sight is offset angularly by a second angle from the normal to theoptically visible pattern, wherein a magnitude of the first angle is thesame as a magnitude of the second angle. In some embodiments, the firstline of sight is offset angularly by a first angle from a normal to theoptically visible pattern and the second line of sight is offsetangularly by a second angle from the normal to the optically visiblepattern, wherein a magnitude of the first angle is different than amagnitude of the second angle. In some embodiments, the ellipticalshapes are configured to appear circular when viewed along a line ofsight separated from a normal to the optically visible pattern by 15degrees. In some embodiments, the reference shape is a triangle. In someembodiments, the reference shape is an equilateral triangle. In someembodiments, the reference shape is an equilateral triangle having aside length of 0.5 inches. In some embodiments, each reference point isno more than 0.5 inches away from another reference point. In someembodiments, each reference point is defined by a centroid of areference point locator. In some embodiments, the reference shape is atriangle configured to appear as an equilateral triangle when viewedalong the first and second lines of sight when the object being trackedis in a home position. In some embodiments, the reference shape is notoptically visible. In some embodiments, the reference shape is a virtualreference shape with the reference points defining vertex points of thevirtual reference shape. In some embodiments, the tracking engine isconfigured to take no longer than 8 milliseconds to determine the poseof the object. In some embodiments, the tracking engine is configured totake no longer than 10 milliseconds to determine the pose of the object.In some embodiments, the tracking engine is configured to repeatedlydetermine the pose of the object at a rate of at least 100 Hz. In someembodiments, the tracking engine is configured to repeatedly determinethe pose of the object at a rate no slower than a scan rate of themedical imaging scanner. In some embodiments, the first and secondoptical detectors are configured to be affixed to the medical imagingscanner. In some embodiments, the first and second optical detectors areconfigured to be affixed to a head cage. In some embodiments, at leastone of the first and second lines of sight is an indirect line of sight.In some embodiments, the indirect line of sight is redirected using amirror. In some embodiments, the indirect line of sight is redirectedusing a prism. In some embodiments, the system further comprises: alight configured to illuminate the optically visible pattern. In someembodiments, the object being tracked is the head of a human patient. Insome embodiments, the first and second optical detectors are configuredsuch that the optically visible pattern is always within a field of viewof the detectors when the optical marker moves within an anticipatedrange of motion. In some embodiments, the tracking engine is furtherconfigured to define a region of interest for each of the digitalimages, the region of interest comprising a portion of the digital imagewithin which the optically visible pattern is anticipated to appear,wherein the tracking engine is configured to analyze only the region ofinterest in determining the pose of the object. In some embodiments, thetracking engine is configured to automatically move the region ofinterest based on anticipated future motion of the object being tracked.In some embodiments, analyzing the first and second images comprises:determining two-dimensional positions of the reference points incoordinate systems of the optical detectors; calculating six baselineattributes of the reference shape based on the two-dimensional positionsof the reference points; estimating, iteratively, a pose of the objectbeing tracked, until an amount of error is within a threshold amount,the amount of error being between the six baseline attributes and sixcomparison attributes, the six comparison attributes calculated byassuming the object is in an estimated pose. In some embodiments, thebaseline and comparison attributes comprise at least one of thefollowing: a sum of displacements of the reference shape as viewed alongthe first line of sight and as viewed along the second line of sight; adifference between displacements of the reference shape as viewed alongthe first line of sight and as viewed along the second line of sight; adifference in distance from a first reference point to a secondreference point as viewed along the first line of sight and as viewedalong the second line of sight; a sum of an apparent median tilt of thereference shape as viewed along the first line of sight and as viewedalong the second line of sight; and a difference between an apparentmedian tilt of the reference shape as viewed along the first line ofsight and as viewed along the second line of sight. In some embodiments,the controller interface is further configured to convert the pose ofthe object from a tracking coordinate system to a scanner coordinatesystem.

In certain embodiments, a motion compensation system for tracking andcompensating for patient motion during a medical imaging scan comprises:an optical marker comprising an optically visible pattern and a mountingportion, the mounting portion configured to affix the optical marker toan object being tracked, the optically visible pattern comprising aplurality of reference point locators, each locator configured to definea single reference point of a reference shape; an optical detectorpositioned to digitally image the optically visible pattern along afirst line of sight and along a second line of sight, the first andsecond lines of sight created by at least a beam splitter, the opticaldetector configured to generate a digital image of the optically visiblepattern, the digital image comprising views of the optically visiblepattern from along both the first and second lines of sight; a trackingengine configured to determine a pose of the object in six degrees offreedom by analyzing the digital image; and a controller interfaceconfigured to generate tracking information based on the pose and toelectronically transmit the tracking information to a scanner controllerto enable compensation within a medical imaging scanner for objectmotion; wherein the tracking engine and controller interface comprise acomputer processor and an electronic storage medium.

In certain embodiments, a computer-implemented method of tracking motionof an object comprises: receiving, by a computer system, from a firstoptical detector a first digital image of an optical marker, the firstdigital image representing a view of the optical marker along a firstline of sight; receiving, by the computer system, from a second opticaldetector a second digital image of the optical marker, the seconddigital image representing a view of the optical marker along a secondline of sight, the second line of sight being different than the firstline of sight, wherein the optical marker comprises a plurality ofoptically visible landmarks, the plurality of optically visiblelandmarks defining a plurality of reference points, the plurality ofreference points defining a reference shape; determining, by thecomputer system, positions of the plurality of reference points in thefirst digital image; determining, by the computer system, positions ofthe plurality of reference points in the second digital image;calculating, using the computer system, a plurality of baselineattributes related to the reference shape, the plurality of baselineattributes calculated based on the determined positions of the pluralityof reference points; and estimating iteratively, using the computersystem, a three-dimensional pose of the object, until a measure of erroris within an acceptable amount, the measure of error calculated based onthe plurality of baseline attributes as compared to a plurality ofcomparison attributes, the plurality of comparison attributes calculatedby assuming the object is in an estimated pose, wherein the computersystem comprises a computer processor and an electronic storage medium.

In some embodiments, each of the plurality of baseline attributes iscalculated based on properties of the reference shape as viewed alongthe first line of sight and the reference shape as viewed along thesecond line of sight. In some embodiments, each of the plurality ofbaseline attributes describes a different variation between thereference shape as viewed along the first line of sight and thereference shape as viewed along the second line of sight. In someembodiments, the plurality of baseline attributes and the plurality ofcomparison attributes comprise at least one of the following: a sum ofdisplacements of the reference shape as viewed along the first line ofsight and as viewed along the second line of sight; a difference betweendisplacements of the reference shape as viewed along the first line ofsight and as viewed along the second line of sight; a difference indistance from a first reference point to a second reference point asviewed along the first line of sight and as viewed along the second lineof sight; a sum of an apparent median tilt of the reference shape asviewed along the first line of sight and as viewed along the second lineof sight; and a difference between an apparent median tilt of thereference shape as viewed along the first line of sight and as viewedalong the second line of sight. In some embodiments, the plurality ofbaseline attributes and the plurality of comparison attributes eachcomprise six attributes. In some embodiments, determining positions ofthe plurality of reference points in the first and second digital imagescomprises: determining a first coordinate set defining a firsttwo-dimensional projection of the reference shape as viewed by the firstoptical detector; and determining a second coordinate set defining asecond two-dimensional projection of the reference shape as viewed bythe second optical detector. In some embodiments, calculating theplurality of baseline attributes comprises comparing the firstprojection to the second projection. In some embodiments, the methodfurther comprises: generating, by the computer system, motion trackingdata based on the estimated three-dimensional pose of the object; andtransmitting electronically to a scanner controller the motion trackingdata to enable a scanner to compensate for motion of the object. In someembodiments, the method takes no longer than 10 milliseconds. In someembodiments, the method takes no longer than 8 milliseconds. In someembodiments, the method further comprises repeating the method at a rateof at least 100 Hz. In some embodiments, the plurality of opticallyvisible landmarks comprise alternative dark and light elliptical shapes.In some embodiments, the elliptical shapes are configured to appearcircular when viewed along the first and second lines of sight when theobject is in a home position. In some embodiments, the reference shapeis a triangle. In some embodiments, the reference shape is anequilateral triangle. In some embodiments, the reference shape is atriangle configured to appear as an equilateral triangle when viewedalong the first and second lines of sight when the object is in a homeposition. In some embodiments, the reference shape is not opticallyvisible. In some embodiments, the reference shape is a virtual referenceshape with the reference points defining vertex points of the virtualreference shape. In some embodiments, each reference point is defined bya centroid of an optically visible landmark.

Certain embodiments comprise a computer readable, non-transitory storagemedium having a computer program stored thereon for causing a suitablyprogrammed computer system to process by one or more processorscomputer-program code by performing a method of tracking motion of anobject when the computer program is executed on the suitably programmedcomputer system, the method comprising: receiving, by a computer system,from a first optical detector a first digital image of an opticalmarker, the first digital image representing a view of the opticalmarker along a first line of sight; receiving, by the computer system,from a second optical detector a second digital image of the opticalmarker, the second digital image representing a view of the opticalmarker along a second line of sight, the second line of sight beingdifferent than the first line of sight, wherein the optical markercomprises a plurality of optically visible landmarks, the plurality ofoptically visible landmarks defining a plurality of reference points,the plurality of reference points defining a reference shape;determining, by the computer system, positions of the plurality ofreference points in the first digital image; determining, by thecomputer system, positions of the plurality of reference points in thesecond digital image; calculating, using the computer system, aplurality of baseline attributes related to the reference shape, theplurality of baseline attributes calculated based on the determinedpositions of the plurality of reference points; and estimatingiteratively, using the computer system, a three-dimensional pose of theobject, until a measure of error is within an acceptable amount, themeasure of error calculated based on the plurality of baselineattributes as compared to a plurality of comparison attributes, theplurality of comparison attributes calculated by assuming the object isin an estimated pose, wherein the computer system comprises a computerprocessor and an electronic storage medium.

In some embodiments, each of the plurality of baseline attributes iscalculated based on properties of the reference shape as viewed alongthe first line of sight and the reference shape as viewed along thesecond line of sight. In some embodiments, each of the plurality ofbaseline attributes describes a different variation between thereference shape as viewed along the first line of sight and thereference shape as viewed along the second line of sight. In someembodiments, the plurality of baseline attributes and the plurality ofcomparison attributes comprise at least one of the following: a sum ofdisplacements of the reference shape as viewed along the first line ofsight and as viewed along the second line of sight; a difference betweendisplacements of the reference shape as viewed along the first line ofsight and as viewed along the second line of sight; a difference indistance from a first reference point to a second reference point asviewed along the first line of sight and as viewed along the second lineof sight; a sum of an apparent median tilt of the reference shape asviewed along the first line of sight and as viewed along the second lineof sight; and a difference between an apparent median tilt of thereference shape as viewed along the first line of sight and as viewedalong the second line of sight. In some embodiments, the plurality ofbaseline attributes and the plurality of comparison attributes eachcomprise six attributes. In some embodiments, determining positions ofthe plurality of reference points in the first and second digital imagescomprises: determining a first coordinate set defining a firsttwo-dimensional projection of the reference shape as viewed by the firstoptical detector; and determining a second coordinate set defining asecond two-dimensional projection of the reference shape as viewed bythe second optical detector. In some embodiments, calculating theplurality of baseline attributes comprises comparing the firstprojection to the second projection. In some embodiments, the methodfurther comprises: generating, by the computer system, motion trackingdata based on the estimated three-dimensional pose of the object; andtransmitting electronically to a scanner controller the motion trackingdata to enable a scanner to compensate for motion of the object. In someembodiments, the method takes no longer than 10 milliseconds. In someembodiments, the method takes no longer than 8 milliseconds. In someembodiments, the method further comprises repeating the method at a rateof at least 100 Hz. In some embodiments, the plurality of opticallyvisible landmarks comprise alternative dark and light elliptical shapes.In some embodiments, the elliptical shapes are configured to appearcircular when viewed along the first and second lines of sight when theobject is in a home position. In some embodiments, the reference shapeis a triangle. In some embodiments, the reference shape is anequilateral triangle. In some embodiments, the reference shape is atriangle configured to appear as an equilateral triangle when viewedalong the first and second lines of sight when the object is in a homeposition. In some embodiments, the reference shape is not opticallyvisible. In some embodiments, the reference shape is a virtual referenceshape with the reference points defining vertex points of the virtualreference shape. In some embodiments, each reference point is defined bya centroid of an optically visible landmark.

In certain embodiments, a system for tracking motion of an objectcomprises: a marker location filter for determining locations of markerreference points, the marker location filter configured to: receive froma first optical detector a first digital image of an optical marker, thefirst digital image representing a view of the optical marker along afirst line of sight; receive from a second optical detector a seconddigital image of the optical marker, the second digital imagerepresenting a view of the optical marker along a second line of sight,the second line of sight being different than the first line of sight,wherein the optical marker comprises a plurality of optically visiblelandmarks, the plurality of optically visible landmarks defining aplurality of reference points, the plurality of reference pointsdefining a reference shape; determine positions of the plurality ofreference points in the first digital image; and determine positions ofthe plurality of reference points in the second digital image; an objectorientation filter for estimating a three-dimensional pose of theobject, the object orientation filter configured to: calculate aplurality of baseline attributes related to the reference shape, theplurality of baseline attributes calculated based on the determinedpositions of the plurality of reference points; and estimate iterativelythe three-dimensional pose of the object, until a measure of error iswithin an acceptable amount, the measure of error calculated based onthe plurality of baseline attributes as compared to a plurality ofcomparison attributes, the plurality of comparison attributes calculatedby assuming the object is in an estimated pose; and one or morecomputers configured to operate the marker location filter and objectorientation filter, wherein the one or more computers comprises acomputer processor and an electronic storage medium.

In some embodiments, each of the plurality of baseline attributes iscalculated based on properties of the reference shape as viewed alongthe first line of sight and the reference shape as viewed along thesecond line of sight. In some embodiments, each of the plurality ofbaseline attributes describes a different variation between thereference shape as viewed along the first line of sight and thereference shape as viewed along the second line of sight. In someembodiments, the plurality of baseline attributes and the plurality ofcomparison attributes comprise at least one of the following: a sum ofdisplacements of the reference shape as viewed along the first line ofsight and as viewed along the second line of sight; a difference betweendisplacements of the reference shape as viewed along the first line ofsight and as viewed along the second line of sight; a difference indistance from a first reference point to a second reference point asviewed along the first line of sight and as viewed along the second lineof sight; a sum of an apparent median tilt of the reference shape asviewed along the first line of sight and as viewed along the second lineof sight; and a difference between an apparent median tilt of thereference shape as viewed along the first line of sight and as viewedalong the second line of sight. In some embodiments, the plurality ofbaseline attributes and the plurality of comparison attributes eachcomprise six attributes. In some embodiments, determining positions ofthe plurality of reference points in the first and second digital imagescomprises: determining a first coordinate set defining a firsttwo-dimensional projection of the reference shape as viewed by the firstoptical detector; and determining a second coordinate set defining asecond two-dimensional projection of the reference shape as viewed bythe second optical detector. In some embodiments, calculating theplurality of baseline attributes comprises comparing the firstprojection to the second projection. In some embodiments, the systemfurther comprises: a controller interface configured to: generate motiontracking data based on the estimated three-dimensional pose of theobject; and transmit electronically to a scanner controller the motiontracking data to enable a scanner to compensate for motion of theobject. In some embodiments, the system is configured to produce theestimated three-dimensional pose of the object in no longer than 10milliseconds. In some embodiments, the system is configured to producethe estimated three-dimensional pose of the object in no longer than 8milliseconds. In some embodiments, the object orientation filter isconfigured to repeatedly estimate the three-dimensional pose of theobject at a rate of at least 100 Hz. In some embodiments, the pluralityof optically visible landmarks comprise alternative dark and lightelliptical shapes. In some embodiments, the elliptical shapes areconfigured to appear circular when viewed along the first and secondlines of sight when the object is in a home position. In someembodiments, the reference shape is a triangle. In some embodiments, thereference shape is an equilateral triangle. In some embodiments, thereference shape is a triangle configured to appear as an equilateraltriangle when viewed along the first and second lines of sight when theobject is in a home position. In some embodiments, the reference shapeis not optically visible. In some embodiments, the reference shape is avirtual reference shape with the reference points defining vertex pointsof the virtual reference shape. In some embodiments, each referencepoint is defined by a centroid of an optically visible landmark.

In certain embodiments, a computer-implemented method of tracking andcompensating for motion of a patient during a medical imaging scancomprises: analyzing, by a computer system, a first digital image of anoptical marker as viewed along a first line of sight, to determine afirst set of two-dimensional coordinates of a plurality of referencepoints as viewed along the first line of sight; analyzing, by thecomputer system, a second digital image of the optical marker as viewedalong a second line of sight, to determine a second set oftwo-dimensional coordinates of the plurality of reference points asviewed along the second line of sight; calculating, by the computersystem, a first set of six principal quantities based on comparingattributes of the first and second sets of two-dimensional coordinates;providing, by the computer system, an estimate of an orientation of thepatient in three dimensions, the estimate comprising six degrees offreedom; calculating, using the computer system, a second set of sixprincipal quantities based on comparing attributes of theoreticalreference points as viewed along the first and second lines of sightwhen the patient is theoretically oriented according to the estimate;determining a quantity of error between the first set of six principalquantities and the second set of six principal quantities; repeatedlymodifying the estimate, calculating the second set of six principalquantities, and determining the quantity of error if the quantity oferror is not within an acceptable threshold and until the quantity oferror is within the acceptable threshold; generating, by the computersystem, motion tracking data based on the estimate of the orientation ofthe patient; and transmitting electronically the motion tracking data toa scanner controller to enable compensating a medical imaging scannerfor motion of the patient during the medical imaging scan, wherein thecomputer system comprises a computer processor and an electronic storagemedium.

In some embodiments, the first set of six principal quantities iscalculated based on properties of a reference shape formed by theplurality of reference points as viewed along the first line of sightand as viewed along the second line of sight. In some embodiments, eachof the principal quantities of the first and second sets of sixprincipal quantities describes a different variation between a referenceshape formed by the plurality of reference points as viewed along thefirst line of sight and as viewed along the second line of sight. Insome embodiments, the principal quantities of the first and second setsof six principal quantities comprise at least one of the following: asum of displacements of a reference shape formed by the plurality ofreference points as viewed along the first line of sight and as viewedalong the second line of sight; a difference between displacements of areference shape formed by the plurality of reference points as viewedalong the first line of sight and as viewed along the second line ofsight; a difference in distance from a first reference point to a secondreference point as viewed along the first line of sight and as viewedalong the second line of sight; a sum of an apparent median tilt of areference shape formed by the plurality of reference points as viewedalong the first line of sight and as viewed along the second line ofsight; and a difference between an apparent median tilt of a referenceshape formed by the plurality of reference points as viewed along thefirst line of sight and as viewed along the second line of sight. Insome embodiments, the method takes no longer than 10 milliseconds. Insome embodiments, the method takes no longer than 8 milliseconds. Insome embodiments, the method further comprises repeating the method at arate of at least 100 Hz. In some embodiments, a reference shape formedby the plurality of reference points is a triangle. In some embodiments,a reference shape formed by the plurality of reference points is anequilateral triangle. In some embodiments, a reference shape formed bythe plurality of reference points is a triangle configured to appear asan equilateral triangle when viewed along the first and second lines ofsight when the patient is in a home orientation. In some embodiments, areference shape formed by the plurality of reference points is notoptically visible. In some embodiments, a reference shape formed by theplurality of reference points is a virtual reference shape with theplurality of reference points defining vertex points of the virtualreference shape. In some embodiments, each reference point is defined bya centroid of an optically visible landmark.

Certain embodiments comprise a computer readable, non-transitory storagemedium having a computer program stored thereon for causing a suitablyprogrammed computer system to process by one or more processorscomputer-program code by performing a method of tracking andcompensating for motion of a patient during a medical imaging scan whenthe computer program is executed on the suitably programmed computersystem, the method comprising: analyzing, by a computer system, a firstdigital image of an optical marker as viewed along a first line ofsight, to determine a first set of two-dimensional coordinates of aplurality of reference points as viewed along the first line of sight;analyzing, by the computer system, a second digital image of the opticalmarker as viewed along a second line of sight, to determine a second setof two-dimensional coordinates of the plurality of reference points asviewed along the second line of sight; calculating, by the computersystem, a first set of six principal quantities based on comparingattributes of the first and second sets of two-dimensional coordinates;providing, by the computer system, an estimate of an orientation of thepatient in three dimensions, the estimate comprising six degrees offreedom; calculating, using the computer system, a second set of sixprincipal quantities based on comparing attributes of theoreticalreference points as viewed along the first and second lines of sightwhen the patient is theoretically oriented according to the estimate;determining a quantity of error between the first set of six principalquantities and the second set of six principal quantities; repeatedlymodifying the estimate, calculating the second set of six principalquantities, and determining the quantity of error if the quantity oferror is not within an acceptable threshold and until the quantity oferror is within the acceptable threshold; generating, by the computersystem, motion tracking data based on the estimate of the orientation ofthe patient; and transmitting electronically the motion tracking data toa scanner controller to enable compensating a medical imaging scannerfor motion of the patient during the medical imaging scan, wherein thecomputer system comprises a computer processor and an electronic storagemedium.

In some embodiments, the first set of six principal quantities iscalculated based on properties of a reference shape formed by theplurality of reference points as viewed along the first line of sightand as viewed along the second line of sight. In some embodiments, eachof the principal quantities of the first and second sets of sixprincipal quantities describes a different variation between a referenceshape formed by the plurality of reference points as viewed along thefirst line of sight and as viewed along the second line of sight. Insome embodiments, the principal quantities of the first and second setsof six principal quantities comprise at least one of the following: asum of displacements of a reference shape formed by the plurality ofreference points as viewed along the first line of sight and as viewedalong the second line of sight; a difference between displacements of areference shape formed by the plurality of reference points as viewedalong the first line of sight and as viewed along the second line ofsight; a difference in distance from a first reference point to a secondreference point as viewed along the first line of sight and as viewedalong the second line of sight; a sum of an apparent median tilt of areference shape formed by the plurality of reference points as viewedalong the first line of sight and as viewed along the second line ofsight; and a difference between an apparent median tilt of a referenceshape formed by the plurality of reference points as viewed along thefirst line of sight and as viewed along the second line of sight. Insome embodiments, the method takes no longer than 10 milliseconds. Insome embodiments, the method takes no longer than 8 milliseconds. Insome embodiments, the method further comprises repeating the method at arate of at least 100 Hz. In some embodiments, a reference shape formedby the plurality of reference points is a triangle. In some embodiments,a reference shape formed by the plurality of reference points is anequilateral triangle. In some embodiments, a reference shape formed bythe plurality of reference points is a triangle configured to appear asan equilateral triangle when viewed along the first and second lines ofsight when the patient is in a home orientation. In some embodiments, areference shape formed by the plurality of reference points is notoptically visible. In some embodiments, a reference shape formed by theplurality of reference points is a virtual reference shape with theplurality of reference points defining vertex points of the virtualreference shape. In some embodiments, each reference point is defined bya centroid of an optically visible landmark.

In certain embodiments, a motion tracking system for tracking andcompensating for motion of a patient during a medical imaging scancomprises: a marker location filter configured to: analyze a firstdigital image of an optical marker as viewed along a first line of sightto determine a first set of two-dimensional coordinates of a pluralityof reference points as viewed along the first line of sight; and analyzea second digital image of the optical marker as viewed along a secondline of sight to determine a second set of two-dimensional coordinatesof the plurality of reference points as viewed along the second line ofsight; an object orientation filter configured to: calculate a first setof six principal quantities based on comparing attributes of the firstand second sets of two-dimensional coordinates; provide an estimate ofan orientation of the patient in three dimensions, the estimatecomprising six degrees of freedom; calculate a second set of sixprincipal quantities based on comparing attributes of theoreticalreference points as viewed along the first and second lines of sightwhen the patient is theoretically oriented according to the estimate;determine a quantity of error between the first set of six principalquantities and the second set of six principal quantities; andrepeatedly modify the estimate, calculate the second set of sixprincipal quantities, and determine the quantity of error if thequantity of error is not within an acceptable threshold and until thequantity of error is within the acceptable threshold; a controllerinterface configured to: generate motion tracking data based on theestimate of the orientation of the patient; and transmit electronicallythe motion tracking data to a scanner controller to enable compensatinga medical imaging scanner for motion of the patient during the medicalimaging scan; and one or more computers configured to operate the markerlocation filter, object orientation filter, and controller interface,wherein the one or more computers comprises a computer processor and anelectronic storage medium.

In some embodiments, the first set of six principal quantities iscalculated based on properties of a reference shape formed by theplurality of reference points as viewed along the first line of sightand as viewed along the second line of sight. In some embodiments, eachof the principal quantities of the first and second sets of sixprincipal quantities describes a different variation between a referenceshape formed by the plurality of reference points as viewed along thefirst line of sight and as viewed along the second line of sight. Insome embodiments, the principal quantities of the first and second setsof six principal quantities comprise at least one of the following: asum of displacements of a reference shape formed by the plurality ofreference points as viewed along the first line of sight and as viewedalong the second line of sight; a difference between displacements of areference shape formed by the plurality of reference points as viewedalong the first line of sight and as viewed along the second line ofsight; a difference in distance from a first reference point to a secondreference point as viewed along the first line of sight and as viewedalong the second line of sight; a sum of an apparent median tilt of areference shape formed by the plurality of reference points as viewedalong the first line of sight and as viewed along the second line ofsight; and a difference between an apparent median tilt of a referenceshape formed by the plurality of reference points as viewed along thefirst line of sight and as viewed along the second line of sight. Insome embodiments, the system is configured to produce the estimate ofthe orientation of the patient in no longer than 10 milliseconds. Insome embodiments, the system is configured to produce the estimate ofthe orientation of the patient in no longer than 8 milliseconds. In someembodiments, the system is configured to repeatedly produce estimates ofthe orientation of the patient at a rate of at least 100 Hz. In someembodiments, a reference shape formed by the plurality of referencepoints is a triangle. In some embodiments, a reference shape formed bythe plurality of reference points is an equilateral triangle. In someembodiments, a reference shape formed by the plurality of referencepoints is a triangle configured to appear as an equilateral trianglewhen viewed along the first and second lines of sight when the patientis in a home orientation. In some embodiments, a reference shape formedby the plurality of reference points is not optically visible. In someembodiments, a reference shape formed by the plurality of referencepoints is a virtual reference shape with the plurality of referencepoints defining vertex points of the virtual reference shape. In someembodiments, each reference point is defined by a centroid of anoptically visible landmark.

In certain embodiments, an optical marker for use in a motion trackingsystem comprises: an optically visible pattern comprising a plurality ofreference point locators, each reference point locator configured todefine a single reference point of a reference shape, each referencepoint locator comprising alternating dark and light elliptical shapescentered on the single reference point; and a mounting portionconfigured to affix the optical marker to an object being tracked.

In some embodiments, the elliptical shapes are configured to appearcircular when viewed along a line of sight other than normal to theoptically visible pattern. In some embodiments, the reference shape is atriangle. In some embodiments, the reference shape is an equilateraltriangle. In some embodiments, the reference shape is an equilateraltriangle having a side length of 0.5 inches. In some embodiments, eachreference point is no more than 0.5 inches away from another referencepoint. In some embodiments, each reference point is defined by acentroid of a reference point locator. In some embodiments, thereference shape is a triangle configured to appear as an equilateraltriangle when viewed along a line of sight other than normal to theoptically visible pattern. In some embodiments, the reference shape isnot optically visible. In some embodiments, the reference shape is avirtual reference shape with the reference points defining vertex pointsof the virtual reference shape. In some embodiments, the marker furthercomprises an internal light source to illuminate the optically visiblepattern. In some embodiments, the mounting portion is configured torigidly affix the optical marker to the object being tracked. In someembodiments, the mounting portion is configured to affix the opticalmarker to top teeth of a patient. In some embodiments, the mountingportion is configured to be custom-fitted to the top teeth of thepatient. In some embodiments, the mounting portion comprises anadhesive. In some embodiments, the optical marker is configured to beutilized as a marker for a motion compensation system for compensatingfor motion during a medical imaging scan. In some embodiments, theoptical marker is configured to be utilized as a marker for a motioncompensation system for compensating for motion during a radiationtherapy process.

In certain embodiments, a system for tracking a moving target having upto six degrees of freedom and rapidly determining positions of themoving target comprises: an optical target fixed to the moving target,the optical target defining a target point; at least two cameraspositioned so as to view the optical target from different directionswith each of the at least two cameras being adapted to recordtwo-dimensional images of the optical target; and a computer processorprogrammed to determine a target position in six degrees of freedomutilizing an algorithm configured to: identify the target point on theoptical target and x, y and z displacement of the target point based onoptical images collected by the at least two cameras; utilize aniteration procedure whereby an approximate first-order solution isproposed and tested against the identified target point to determineresidual errors which are then divided by local derivatives with respectto each component of rotation and translation, to determine an iterativecorrection; repeat the iteration procedure until residual error becomessmaller than a desired accuracy; and utilize the results of the repeatediteration procedure to determine the target position at a rate of atleast 100 times per second.

In some embodiments, the moving target is a human head. In someembodiments, the system is configured to interface as a component of anMRI device. In some embodiments, the iteration procedure is a variant ofthe Newton-Raphson method. In some embodiments, movements are measuredrelative to a pivot point in a patient's neck. In some embodiments,measurements are updated at a rate of at least 100 solutions per secondwith a latency of less than 10 milliseconds. In some embodiments,measurements are updated at a rate of at least 200 solutions per secondwith a latency of less than 10 milliseconds. In some embodiments, thesystem is adapted to report to MRI systems a position of a patient'shead with accuracies better than 0.1 mm in distances and 0.1 degree inangles. In some embodiments, the optical target comprises at least threeconcentric sub-targets. In some embodiments, the concentric sub-targetscomprise dark concentric sub targets on a high contrast diffuselyreflective background. In some embodiments, the concentric sub-targetscomprise high contrast diffusely reflective concentric sub-targets on adark background. In some embodiments, the concentric sub-targetscomprise dark concentric sub-targets on a high contrast retro-reflectivebackground. In some embodiments, the concentric sub-targets comprisehigh contrast retro-reflective concentric sub-targets on a darkbackground. In some embodiments, the optical target is fixed to at leastone of a patient's upper teeth. In some embodiments, the threeconcentric sub-targets are each concentric ellipses. In someembodiments, the computer processor is programmed to calculate centroidsof each sub-target by dividing (a) a sum of a product of pixel intensityand pixel position by (b) a sum of pixel intensity in a sub-pixel array.In some embodiments, the pixel array is about 48×48 pixels.

In some embodiments, the iteration procedure comprises: locatingpositions of three target centroids on each of the two cameras andcalculating six principal quantities: Σ_(HD) Δ_(HD) Σ_(VD) Δ_(BL) Σ_(MT)Δ_(MT); making an initial guess of (0, 0, 0, 0, 0, 0) for the subjectdisplacements and rotations (φ, θ, ψ, Δx, Δy, Δz) leading to thosecentroid positions; entering the guess values for (φ, θ, ψ, Δx, Δy, Δz)into a translation matrix and calculating a corresponding translated 3-Dtarget position (x_(⊥)i,y_(⊥)i,z_(⊥)i) for each of the three targetcentroids; calculating a position (horizontal and vertical pixel number)of a projection of each of the three target centroids located on eachcamera system and calculating six principal quantities using this datafrom the two cameras: Σ_(HD) Δ_(HD) Σ_(VD) Δ_(BL) Σ_(MT) Δ_(MT);comparing these six calculated principal quantities with measured valuesfrom locating the positions of the three target centroids; listing thedifferences as a matrix of errors in the guessed/calculated quantitiesrelative to the measured quantities: (σ_(Σ) _(HD) , σ_(Δ) _(HD) , σ_(Σ)_(VD) , σ_(Δ) _(BL) , σ_(Σ) _(MT) , σ_(Δ) _(MT) ); determining localpartial derivatives of: Σ_(HD) Δ_(HD) Σ_(VD) Δ_(BL) Σ_(MT) Δ_(MT) byrepeating the iteration procedure six times with small addeddisplacements in each degree of freedom, one at a time; determining acoarse correction matrix by dividing the error matrix by the derivativematrix, to improve the initial guess and reduce the residual error;creating a better displacement matrix:

[φ+Δφ,θ+Δθ,ψ+Δψ,Δx+Δ(Δx),Δy+Δ(Δy),Δz+Δ(Δz)];

repeating a portion of the iteration procedure for a second and finaliteration, starting with the coarse-corrected displacement matrix as theguess value; after each successive camera frame increment, repeating theiteration procedure, but using the result of the second and finaliteration for the guess value.

In certain embodiments, a computer-implemented method for tracking amoving target having up to six degrees of freedom and rapidlydetermining positions of the target comprises: attaching an opticaltarget to the moving target; positioning at least two cameras positionedso as to view the optical target from different directions with each ofthe at least two cameras being adapted to record two dimensional imagesof the optical target defining a target point; programming a computerprocessor to determine a target position in six degrees of freedomutilizing an algorithm adapted to: identify the target point on theoptical target and the x, y and z displacement of the target point basedon optical images collected by the at least two cameras; utilize aniteration procedure whereby an approximate first-order solution isproposed and tested against the identified target point to determineresidual errors which are then divided by local derivatives with respectto each component of rotation and translation, to determine an iterativecorrection; repeat the iteration procedure until residual error becomessmaller than a desired accuracy, and utilize the results of the repeatediteration procedure to determine the target position at a rate of atleast 100 times per second.

In some embodiments, the target is a human head. In some embodiments,tracking results are utilized as an input to an MRI device so as toadjust its magnetic fields to compensate for movements of the humanhead. In some embodiments, the iteration procedure is a variant of theNewton-Raphson method. In some embodiments, movements are measuredrelative to a pivot point in a patient's neck. In some embodiments,measurements are updated at a rate of at least 100 solutions per secondwith a latency of less than 10 milliseconds. In some embodiments,measurements are updated at a rate of at least 200 solutions per secondwith a latency of less than 10 milliseconds. In some embodiments, thesystem is adapted to report to MRI systems the position of the head withaccuracies better than 0.1 mm in distances and 0.1 degree in angles. Insome embodiments, the optical target comprises at least three concentricsub-targets. In some embodiments, the concentric sub-targets comprisedark concentric sub targets on a high contrast diffusely reflectivebackground. In some embodiments, the concentric sub-targets comprisehigh contrast diffusely reflective concentric sub-targets on a darkbackground. In some embodiments, the concentric sub-targets comprisedark concentric sub-targets on a high contrast retro-reflectivebackground. In some embodiments, the concentric sub-targets comprisehigh contrast retro-reflective concentric sub-targets on a darkbackground. In some embodiments, the three concentric sub-targets areeach concentric ellipses. In some embodiments, the computer processor isprogrammed to calculate centroids of each sub-target by dividing (a) asum of a product of pixel intensity and pixel position by (b) a sum ofpixel intensity in a sub-pixel array. In some embodiments, the pixelarray is about 48×48 pixels.

In some embodiments, the iteration procedure comprises: locatingpositions of three target centroids on each of the two cameras andcalculating six principal quantities: Σ_(HD) Δ_(HD) Σ_(VD) Δ_(BL) Σ_(MT)Δ_(MT); making an initial guess of (0, 0, 0, 0, 0, 0) for the subjectdisplacements and rotations (φ, θ, ψ, Δx, Δy, Δz) leading to thosecentroid positions; entering the guess values for (φ, θ, ψ, Δx, Δy, Δz)into a translation matrix and calculating a corresponding translated 3-Dtarget position (x_(⊥)i,y_(⊥)i,z_(⊥)i) for each of the three targetcentroids; calculating a position (horizontal and vertical pixel number)of a projection of each of the three target centroids located on eachcamera system and calculating six principal quantities using this datafrom the two cameras: Σ_(HD) Δ_(HD) Σ_(VD) Δ_(BL) Σ_(MT) Δ_(MT);comparing these six calculated principal quantities with measured valuesfrom locating the positions of the three target centroids; listing thedifferences as a matrix of errors in the guessed/calculated quantitiesrelative to the measured quantities: (σ_(Σ) _(HD) , σ_(Δ) _(HD) , σ_(Σ)_(VD) , σ_(Δ) _(BL) , σ_(Σ) _(MT) , σ_(Δ) _(MT) ); determining localpartial derivatives of: Σ_(HD) Δ_(HD) Σ_(VD) Δ_(BL) Σ_(MT) Δ_(MT) byrepeating the iteration procedure six times with small addeddisplacements in each degree of freedom, one at a time; determining acoarse correction matrix by dividing the error matrix by the derivativematrix, to improve the initial guess and reduce the residual error;creating a better displacement matrix:

[φ+Δφ,θ+Δθ,ψ+Δψ,Δx+Δ(Δx),Δy+Δ(Δy),Δz+Δ(Δz)];

repeating a portion of the iteration procedure for a second and finaliteration, starting with the coarse-corrected displacement matrix as theguess value; after each successive camera frame increment, repeating theiteration procedure, but using the result of the second and finaliteration for the guess value.

In an embodiment, a system is configured for tracking a moving targethaving up to six degrees of freedom and rapidly determining positions ofthe target, said system includes an easy to locate precision opticaltarget fixed to the target. The system can also include at least twocameras positioned so as to view the optical camera from differentdirections with each of the at least two cameras being adapted to recordtwo dimensional images of the precision optical target defining aprecise target point. In an embodiment, a computer processor isprogrammed to determine the target movement in Cartesian coordinates ofx, y and z and pitch, roll and yaw utilizing an algorithm adapted toidentify a set of precise target points on the precision optical targetand the x, y and z displacement and the pitch, roll and yaw rotation ofthe precise target points based on optical images collected by the atleast two cameras. The system can utilize an iteration procedure wherebyan approximate first-order solution is proposed and tested against theidentified precise target point projections on the cameras to determineresidual errors which are then divided by the local derivatives withrespect to each component of rotation and translation, to determine aniterative correction. The system can be configured to repeat the aboveactions until residual error becomes smaller than desired accuracy.Using this process the system can be configured to determine theposition of the target at rates of at least 100 times per second withtranslations accuracies of about or no more than about 0.1 mm and angleaccuracies of about or no more than about 0.1 degrees. With repetitionrates in the range of 100 times per second, the full 6-DOF movementdetermination can be performed for each repetition. In these embodimentsthe results of each movement determination is used for the initial firstorder solution during the next iteration.

The six degrees of freedom movements are over orthogonal directions x,y, and z and roll, pitch and yaw angles. Direction x is along the spinalaxis. Direction y perpendicular to x is along the shoulder to shoulderdirection and direction z is perpendicular to both x and y and in thefloor-to-ceiling direction assuming the patient is lying on his backparallel to the floor. The roll angle is about the x-axis; the anglemade by a shaking head “No”. The pitch angle is about the y-axis; theangle made by shaking head “Yes” and the Yaw angle is about the z-axis,the angle made by leaning head toward a shoulder.

In an embodiment, the desired accuracy is about 0.1 mm for each of thedirections and about 0.1 degrees for each of the angles. Movements aremeasured relative to a pivot point in the patient's neck. In anembodiment the pivot point is located at the base of the patient's neckwhere the head swivels for nod turn and lean motions. The offset of theprecision optical target from this pivot point position is Δy=0,Δx−4.5″, Δz=5.5″. The precision of these offsets is not critical sinceall motions of interest are relative motions. The six measurements arex, y, and z distances and roll, pitch and yaw angles. In someembodiments, the measurements are up-dated at a rate of about 100solutions per second with a latency of about 10 milliseconds. The systemcan be configured to report to MRI systems the exact position or theapproximate position of the head with accuracies of about or better thanabout 0.1 mm in distances and about 0.1 degree in angles.

One possible coordinate system for reporting 6-DOF motions to the MRIfield compensation system is a Cartesian system aligned with thesymmetry axis of the head coil. The head coil coordinate system iscoincident with body coordinates in the nominal (“square”) headposition. Target displacements and rotations can be reported to the coilfield compensation system using this system of coordinates.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other features, aspects, and advantages of the presentinventions are described in detail below with reference to the drawingsof various embodiments, which are intended to illustrate and not tolimit the inventions. The drawings comprise the following figures inwhich:

FIG. 1A is an embodiment of a schematic diagram illustrating a frontview of a medical imaging scanner as a part of a motion compensationsystem.

FIG. 1B is a schematic diagram illustrating a side view of the medicalimaging scanner as a part of the motion compensation system of FIG. 1A.

FIG. 2A is a front view of an embodiment of an optical marker.

FIG. 2B is a perspective view of the optical marker of FIG. 2A in usewith a patient.

FIG. 2C is a front view of the optical marker of FIG. 2A, the opticalmarker shown next to a U.S. penny for scale.

FIG. 3A is a block diagram depicting an embodiment of a motioncompensation system.

FIG. 3B is a block diagram depicting another embodiment of a motioncompensation system.

FIG. 4A is another embodiment of a schematic diagram illustrating afront view of a medical imaging scanner as part of a motion compensationsystem.

FIG. 4B is a schematic diagram illustrating a side view of the medicalimaging scanner as a part of the motion compensation system of FIG. 4A.

FIG. 4C is another embodiment of a schematic diagram illustrating afront view of a medical imaging scanner as part of a motion compensationsystem.

FIG. 4D is a schematic diagram illustrating a side view of the medicalimaging scanner as a part of the motion compensation system of FIG. 4C.

FIG. 4E is another embodiment of a schematic diagram illustrating afront view of a medical imaging scanner as part of a motion compensationsystem.

FIG. 4F is a schematic diagram illustrating a side view of the medicalimaging scanner as a part of the motion compensation system of FIG. 4E.

FIG. 4G is another embodiment of a schematic diagram illustrating afront view of a medical imaging scanner as part of a motion compensationsystem.

FIG. 4H is another embodiment of a schematic diagram illustrating a sideview of a medical imaging scanner as part of a motion compensationsystem.

FIG. 4I is another embodiment of a schematic diagram illustrating a sideview of a medical imaging scanner as part of a motion compensationsystem.

FIG. 4J is another embodiment of a schematic diagram illustrating a sideview of a medical imaging scanner as part of a motion compensationsystem.

FIG. 4K is another embodiment of a schematic diagram illustrating afront view of a medical imaging scanner as part of a motion compensationsystem.

FIG. 4L is another embodiment of a schematic diagram illustrating amotion compensation system.

FIG. 4M is another embodiment of a schematic diagram illustrating amotion compensation system.

FIG. 4N is another embodiment of a schematic diagram illustrating amotion compensation system.

FIG. 5A is a front perspective view of an embodiment of an opticalmarker connected to a mounting portion.

FIG. 5B is a side perspective view of the optical marker of FIG. 5A.

FIG. 5C is a top view of another embodiment of an optical markerconnected to a mounting portion.

FIG. 5D is a top view of another embodiment of an optical markerconnected to a mounting portion.

FIG. 5E is a perspective view of an embodiment of an optical markerconnected to a patient.

FIG. 6A is a perspective view of an embodiment of an optical markerusing distributed reference points.

FIG. 6B is a perspective view of an embodiment of an optical markerusing distributed reference points and an embodiment of an opticalmarker using integrated reference points.

FIG. 6C is a perspective view of an embodiment of multiple opticalmarkers attached to an object being tracked.

FIG. 7A depicts an embodiment of a process flow diagram illustrating anexample of a motion compensation process.

FIG. 7B depicts another embodiment of a process flow diagramillustrating an example of a motion compensation process.

FIG. 7C depicts another embodiment of a process flow diagramillustrating an example of a motion compensation process.

FIG. 7D depicts another embodiment of a process flow diagramillustrating an example of a motion compensation process.

FIGS. 8A and 8B show how two cameras together provide sensitivity neededto track motion, according to some embodiments of the invention.

FIGS. 9A and 9B show how a patient's head and two cameras are located inan MRI device, according to some embodiments of the invention.

FIGS. 10A and 10B show how Cartesian coordinates are used relative to apatient's head for the purpose of tracking motions, according to someembodiments of the invention.

FIG. 11 shows how three points of the precision optical target areimaged on the focal plane of each of the two cameras, according to someembodiments of the invention.

FIG. 12 shows the results on one camera image of a 15 degree yawmovement (about the z-axis), according to some embodiments of theinvention.

FIGS. 13A and 13B shows how two cameras are able to monitor precisely apitch movement (about the y-axis), according to some embodiments of theinvention.

FIGS. 14A and 14B show how a roll movement (about the x-axis) ismonitored, according to some embodiments of the invention.

FIG. 15 shows how x-axis translation (positive toward the top of thepatient's head) is monitored on one camera, according to someembodiments of the invention.

FIGS. 16A and 16B shows the effect of y-axis translation (positive tothe patient's right side) as monitored on the two cameras, according tosome embodiments of the invention.

FIGS. 17A and 17B show the effect of z-axis translation (toward theceiling), according to some embodiments of the invention.

FIGS. 18A and 18B show the effect of simultaneous pitch and x-axis andz-axis translation, according to some embodiments of the invention.

FIGS. 19A and 19B show the effect of simultaneous roll and y-axis andz-axis translation, according to some embodiments of the invention.

FIGS. 20A and 20B display features of an iteration technique utilized toprecisely monitor head movement utilizing the camera images of theprecision optical target, according to some embodiments of theinvention.

FIG. 20C is a flow diagram of an iterative process for trackingmovement.

FIG. 21 is a block diagram depicting an embodiment of a computer systemconfigured to implement one or more embodiments of the methods, devices,and systems described herein.

FIGS. 22A and 22B show techniques for camera calibration, according tosome embodiments of the invention.

FIG. 23 is an embodiment of a schematic diagram illustrating atherapeutic applications connected to one or more marker trackingsystems.

FIG. 24 is a flowchart depicting an embodiment of a process forcontrolling the application of a therapeutic therapy based on coordinatedata generated from one or more tracking systems.

FIG. 25 is an embodiment of a schematic diagram illustrating atherapeutic applications connected to a marker tracking system.

FIG. 26 a flowchart depicting an embodiment of a process for controllingthe application of a therapeutic therapy based on coordinate datagenerated from a tracking system.

FIGS. 27A-27G illustrate a laboratory configuration testing anembodiment of the concepts described herein.

FIG. 28 is an embodiment of an optical marker.

FIGS. 29A-29B illustrate embodiments of optical markers comprising subtargets on distributed substrates.

FIGS. 30A-30F illustrate various embodiments of optical marker subtargets on individual substrates.

FIG. 31 is an embodiment of a process flow diagram illustrating anexample of a motion tracking process.

FIG. 32 is an embodiment of a process flow diagram illustrating anexample of a marker centroid determination process.

FIG. 33 is an embodiment of a process flow diagram illustrating exampleimages as a part of the process flow illustrated in FIG. 32.

FIG. 34 illustrates an example of a marker or optical tracker thatavoids rotational degeneracy, according to one embodiment.

FIG. 35 illustrates an example of a marker or optical tracker thatavoids rotational degeneracy, according to one embodiment.

FIGS. 36A and 36B illustrate embodiments of sets of uniquelyidentifiable markers.

FIG. 37 is a flow chart that illustrates a marker pattern centroiddetection process, according to one embodiment.

FIG. 38 illustrates how an image of a rotationally asymmetric marker maybe processed in some embodiments in order to locate the centroids in themarker.

FIG. 39 illustrates the orientation and positions of components in a MRsystem, according to one embodiment.

FIG. 40 illustrates one embodiment of a reference target used intracking system calibration.

FIG. 41 is a flow chart that illustrates how the system may adapt to theaddition or removal of cameras.

FIG. 42 is a block diagram depicting an embodiment of a motioncompensation system.

FIG. 43 illustrates nominal angles associated with camera positioning.

FIG. 44 illustrates the focal plane of a camera for a case whenx_(c1)=x₀.

FIG. 45 illustrates a vector between the apparent origin and the pointof intersection of the camera optical axis with the perpendicular planethat contains the target point.

DETAILED DESCRIPTION

Although several embodiments, examples, and illustrations are disclosedbelow, it will be understood by those of ordinary skill in the art thatthe inventions described herein extend beyond the specifically disclosedembodiments, examples, and illustrations and includes other uses of theinventions and obvious modifications and equivalents thereof.Embodiments of the inventions are described with reference to theaccompanying figures, wherein like numerals refer to like elementsthroughout. The terminology used in the description presented herein isnot intended to be interpreted in any limited or restrictive mannersimply because it is being used in conjunction with a detaileddescription of certain specific embodiments of the inventions. Inaddition, embodiments of the inventions can comprise several novelfeatures and no single feature is solely responsible for its desirableattributes or is essential to practicing the inventions hereindescribed.

With the use of diagnostic technologies and therapeutic technologies, itcan be advantageous to track for patient movement with a high degree ofaccuracy. Such high accuracy tracking can improve the imaging qualityobtained and produced by diagnostic equipment, such as imagingtechnologies. Further, the use of high accuracy patient movementtracking technology can improve the application of patient therapies,such as radiation treatment, proton treatment, and the like. Byaccounting for patient movement with a high degree of accuracy,therapeutic technologies can apply therapies only to the targeted tissueand avoid healthy surrounding tissue.

The embodiments disclosed herein relate to a patient motion trackingsystem that can track patient movement with translation accuracies ofabout 0.1 mm and angle accuracies of about 0.1 degrees. As disclosedherein, the system can be configured to utilize a non-stereo approach todetermining the 6 degrees of freedom movement of the patient. In anembodiment, the system can comprise two cameras that are positionedorthogonal and perpendicular on a single plane. In an embodiment, thetwo cameras need not be in a single plane, but are positioned such thatthe two cameras are not viewing the target from generally the samedirection. The system can be configured to compare the appearance of thetarget on one camera with the other camera while not accounting forwhich pixel number they fall on in either camera. By comparing theappearance of the target between the two cameras, the system can beconfigured to extract the 6 degrees of freedom movement on a very smalltarget.

In an embodiment, the system can be configured to extract the movementdata based on analyzing the images of the target from the two cameras inorder to generate a predicted value for at least one of the variables inthe 6 degrees of freedom. For example, the system can be configured toanalyze the image of the target and predict a value for the pitch. Thesystem can then be configured to compare the predicted value to thevalue of the particular variable with that which is shown in the actualimage of the target. The system can be configured to repeat this processusing an iterative approach to continuously improve the predicted valueof one of the variables in the 6 degrees of freedom. The system can beconfigured to perform this iterative process for each variable in the 6degrees of freedom.

The foregoing methodology for tracking patient movement can be appliedin the diagnostic context as well as in the therapeutic context. Forexample, as disclosed herein, the system can be configured to trackpatient movement in order to feed such movement data to an MRI scannersuch that the MRI scanner can adjust the focus and position of thescanner in order to produce a clear MRI image of the patient. Further,the system can be configured to connect to therapeutic technologies. Forexample, the system can be configured to track patient movement in orderto direct a therapeutic radiation beam at a diseased tissue region whileavoiding surrounding healthy tissue.

There are various technologies for therapeutic radiation and othertherapeutics. For example, it can be advantageous in radiation therapy,proton therapy, or other therapies to dynamically apply the radiation toa targeted area in order to account for patient movement. Patientmovement can include respiration, twitches or any other voluntary orinvoluntary movements of the patient. By dynamically and automaticallytracking patient movement, radiation therapy, proton therapy, and anyother kind of therapy can be applied in a more targeted way, therebyallowing surrounding healthy tissue to be avoided and/or unharmed. Thesystems disclosed herein can be adapted and configured to track patienttranslations with accuracies of about 0.1 mm and angle accuracies ofabout 0.1 degrees in order to better apply radiation therapy, protontherapy, or any other therapy to the targeted tissue or area of thebody.

In an embodiment, a system can be configured to utilize optical trackingbased on the methods disclosed herein in order to track patient movementand/or or another device, for example, electronics packages that areconfigured to identify fiducial markers implanted inside a patient. Inan embodiment, the system can be configured to utilize the electronicspackage in order to identify the location of the fiducial markers withinthe patient. By identifying the location of the fiducial markers, thesystem needs to identify the location of the electronics package inorder to determine the location of the fiducial markers with respect toa scanner and/or a therapeutic equipment device.

The patient tracking movement system, disclosed herein, can be utilizedto track periodic involuntary movement of the patient, such asbreathing. By tracking the periodic patient movement with a high degreeof accuracy, the system can be configured to apply a radiation therapy,a proton therapy, or the like during strategic moments when the targettissue is in a certain position while the patient's involuntarymovements continue. Additionally, the system can be configured to tracknot only normal breathing movement of the patient, but also the systemcan be configured to track irregular movement of the patient caused bypatient activity or based on diseased tissue of the patient. Forexample, when a patient is running, the ribs of the patient have alarger egression that the system can track in order to continuouslyidentify a target tissue area. In another example, the patient may besuffering from COPD or other breathing disorder or diagrammatic issues.For example, the patient could be suffering from theurofusion, which iswater outside the lung that prevents the patient from breathing or atumor is irritating a lung region thereby preventing normal breathing.The system can be configured to track such irregular patient movementsdue to such conditions.

In order to apply a therapy, such as radiation therapy, the radiationbeam generator must determine the location of the electronics packagerelative to the beam generator in order to properly direct the radiationtherapy to the targeted tissue. Accordingly, it is necessary to trackthe position of the electronics package relative to the radiation beamgenerator or other therapeutic equipment. It can be advantageous totrack the position of the electronics package with a high degree ofaccuracy in order to better target the desired tissue. In systems wherethe electronics package is configured to track the location of fiducialmarkers implanted within the patient, such systems have two possiblesources of error. One source of error can be derived from tracking theposition of the fiducial markers using the electronics package and thesecond source of error can be derived from tracking the position of theelectronics package relative to the therapeutic equipment generator.Accordingly, it can be advantageous to identify the position of theelectronics package with a high degree of accuracy in order to avoidcompounding the sources of error.

Motion Compensation Systems

FIG. 1A is an embodiment of a schematic diagram illustrating a frontview of a medical imaging scanner 104 as part of a motion compensationsystem 100. FIG. 1B is a schematic diagram illustrating a side view ofthe medical imaging scanner 104 as a part of the motion compensationsystem 100 of FIG. 1A. The motion compensation system 100 can be usedto, for example, track the motion of a patient undergoing a medicalimaging procedure to enable a medical imaging scanner to adjust orotherwise compensate for that motion, to reduce or eliminate motionartifacts in the resulting medical images. The motion compensationsystem 100 illustrated in FIGS. 1A and 1B comprises a motion trackingsystem 102, a scanner 104, a scanner controller 106, two detectors 108,and an optical marker or target 110. In this embodiment, the opticalmarker 110 is shown attached to a patient 112 positioned on a table 114of the medical imaging scanner 104. The scanner 104 can be, for example,a magnetic resonance imaging scanner. The optical marker 110 can beconfigured as further described below, for example as described inreference to FIG. 2A.

In this embodiment, the optical marker 110 is configured to be viewableby each of the two detectors 108. The detectors 108 can be, for example,digital cameras capable of acquiring images of the optical marker 110and transmitting those images to the motion tracking system 102. In thisembodiment, each of the detectors 108 is configured to view the opticalmarker 110 from along a different line of sight. This can be helpful,for example, to enable the motion tracking system 102 to analyze twodimensional images of the optical marker 110 from different vantagepoints to help in locating the optical marker 110 to estimate patientmotion or pose. In this embodiment, the detectors 108 each areconfigured to view the optical marker 110 along a line of sight 120separated from each other by an angle 122. In this embodiment, the angle122 is approximately 90 degrees. Other angles may be used, such as 30degrees, 45 degrees, 60 degrees, 70 degrees, etc. In some embodiments,90 degrees is an optimal angle to enable maximum differentiation of inplane and out of plane motion of the optical marker 110, as furtherdescribed below with reference to FIGS. 8A and 8B. For example, if theoptical marker 110 moves in a direction that is directly along the lineof sight of one detector, that detector may have a harder timedistinguishing motion of the optical marker 110 than the other detector.On the other hand, the other detector may relatively easily detect themotion of the optical marker 110, as the motion is perpendicular to thatdetector's line of sight.

In some embodiments, the angle 122 may be referred to as a scissorangle. In the embodiment illustrated in FIG. 1A, the scissor angle isthe angle at which the detectors 108 are directly viewing the marker110. However, in other embodiments, the scissor angle may be a virtualangle, as the lines of sight from the detectors to the marker may beredirected by mirrors and/or other means, such as beam splitters,prisms, fiber optics, and/or the like. In that case, the scissor angleis the apparent angle at which the detectors are viewing the marker. Forexample, as further described below with reference to FIGS. 4E and 4F,the detectors 108 of the motion compensation system 440 are positionedwith lines of sight collinear to each other near the detectors. However,mirrors are utilized to redirect the lines of sight such that a virtualscissor angle of approximately 90 degrees is accomplished near themarker.

Mirrors or other devices used to redirect a line of sight have bothadvantages and disadvantages. For example, disadvantages of mirrorsinclude that they could potentially vibrate, potentially introducingerror into the object orientation determination process. As anotherexample, the further away a mirror is from a detector, generally thelarger the mirror needs to be to enable an equivalent range of vision.Accordingly, it can be advantageous to position a mirror relativelyclose to a detector to enable the mirror to be relatively small. Oneadvantage of using mirrors or other sight line redirection methods isthat a virtual scissor angle can be configured to be closer to anoptimal scissor angle of 90°, even when a particular medical imagingscanner configuration may not allow for detectors that are positioned todirectly view a marker using a 90° scissor angle. Further, some mirrorsare not conductive, which can be advantageous in magnetic resonanceimaging, because nonconductive mirrors will not introduce artifacts intoMRI images. A digital camera, on the other hand, may include conductivecomponents and/or a wire leading to the detector may include conductivecomponents. When a digital camera and/or its wire are within the medicalimaging envelope, they may introduce artifacts into MRI images.

The embodiment of a motion compensation system 100 illustrated in FIGS.1A-1B is not shown to scale, but is rather show at a scale that helpsfacilitate illustration of the system. Other figures, such as is shownin FIGS. 4A-4K, are also not shown to scale. Additionally, mostembodiments illustrated in these figures and described in thisspecification comprise a motion compensation system operating in realtime or substantially in real time to correct a scanner for motion of apatient or object. However, in other embodiments, a motion compensationsystem can be configured to operate by processing images usingpost-processing after they have been created by a scanner to remove anymotion artifacts.

In the embodiment of a motion compensation system 100 illustrated inFIGS. 1A and 1B, the detectors 108 are positioned at an angle ofapproximately 90 degrees along a transverse axis of the scanner 104, butare positioned at an angle of approximately 0 degrees along alongitudinal axis of the scanner 104, as shown in FIG. 1B. In thisembodiment, the detectors 108 are configured to be positioned directlyabove a nominal home position of the optical marker 110, as shown inFIG. 1B. However, the detectors may be setup in various otherconfigurations, as further described below.

FIGS. 2A-2C illustrate one embodiment of an optical marker 110 used witha motion compensation system. FIG. 2A is a front view of the opticalmarker 110. FIG. 2B is a perspective view of the optical marker 110 ofFIG. 2A in use with a patient 112. FIG. 2C is a front view of theoptical marker of FIG. 2A, the optical marker shown next to a U.S. penny212 for scale.

The optical marker 110 comprises a pattern 202 that defines a referenceshape. In this embodiment, the pattern 202 defines a reference shape ofan equilateral triangle having sides of approximately 0.5 inches. Ateach vertex of the equilateral triangle reference shape is a referencepoint locater 204. In this embodiment, each reference point locator 204comprises a series of alternating black and white (or dark and light)elliptical shapes, with a centroid of the reference point locater 204being positioned at the vertex of the equilateral triangle referenceshape. In various embodiments, different reference shapes can be usedand reference point locators can take various forms, as long as thereference point locators are able to be detected and analyzed by amotion tracking system to determine important points, inflection points,critical points, or vertex points of a reference shape.

In this embodiment, the reference point locators 204 are elliptical inshape and positioned such that they are configured to be visible as acircular pattern from a 45 degree viewing angle. This can beadvantageous, because, when used in a system such as the exampleillustrated in FIGS. 1A and 1B, where the detectors 108 are viewing theoptical marker 110 along a 45 degree sight line, any images of theoptical marker 110 at the nominal home location will illustrate thereference point locators 204 as circles. Then, if the marker is moved,that motion will result in a non-circular elliptical pattern showing upon the images of the optical marker. Further description of the opticalmarker configuration is given below.

FIG. 2B illustrates that the optical marker 110 can, in someembodiments, be configured to mount to a patient's top teeth. This canbe advantageous to retain or affix the optical marker 110 in a rigid orsubstantially rigid location with respect to the patient's skull. Asdescribed below, various methods and devices may be used to attach anoptical marker to a patient or other object of interest.

FIG. 3A is a block diagram depicting an embodiment of a motioncompensation system 300. The motion compensation system 300 can besimilar to, for example, the motion compensation system 100 illustratedin FIGS. 1A and 1B. The motion compensation system 300 comprises twodetectors 108, a motion tracking system 102, a scanner 104, and ascanner controller 106. The scanner 104 is a medical imaging scanner,such as an MRI machine. The scanner controller 106 can be configured tooperate and control the scanner 104, such as by adjusting the scanner104 in real time or substantially real time to correct for motion of theobject or patient being scanned. The scanner controller 106 can adjustfor motion based on motion tracking data or information received fromthe motion tracking system 102.

The detectors 108 can comprise, for example, digital cameras. Althoughin this embodiment there are two detectors 108, various otherembodiments may utilize more or fewer detectors based on theapplication. For example, an embodiment of a motion compensation systemmay comprise more detectors to increase accuracy of motion trackingand/or to add redundancy to a motion compensation system. For example, amotion compensation system may comprise four detectors, with motiontracking being performed only using two of the detectors at any onetime. This may be advantageous, for example, when obstructions may hidean optical marker from view of one or more detectors depending on theposition of the object being tracked.

Although in this embodiment and various other embodiments describedherein the detectors are optical digital cameras, various other motioncompensation systems may utilize detectors other than optical cameras.For example, a detector may be an infrared camera configured to view atarget or marker viewable by an infrared camera. In other embodiments,the detectors may comprise laser detectors, sonar detectors, radardetectors, and various other types of detectors capable of locating amarker and/or creating a two dimensional digital image or representationof a marker.

The motion tracking system 102 comprises a tracking engine 304, acalibration engine 306, a controller interface 308, and a motiondatabase 310. The motion database 310 can be configured to store motiontracking information, such as object pose estimates created by thetracking engine 304. In some embodiments, the motion database 310 can beconfigured to be persistent storage to store the motion information forlater retrieval and usage after the completion of an imaging scan. Insome embodiments, the motion database 310 comprises a short term memoryor buffer or cache to store object pose estimates just temporarily untilthey are included in motion data and sent to the scanner controller 106by the controller interface 308.

The calibration engine 306 can be configured to calibrate the motioncompensation system. The calibration engine 306 comprises a calibrationdatabase 316, a detector calibration filter 318, and a targetcalibration filter 320. In some embodiments, the calibration engine 306can be configured to calibrate the system at initial startup or initialsystem assembly, with calibration not being needed for later operationof the motion compensation system 300. In other embodiments, thecalibration engine 306 is utilized for at least some calibrationprocedures during some or all motion tracking procedures. The detectorcalibration filter 318 can be configured to calibrate the detectors 108to the motion compensation system. In some embodiments, the detectorcalibration filter 318 can be configured to calibrate the detectors byenabling a manufacturer or assembler of the motion compensation systemto input information specific to each detector 108, such as focallength, resolution, etc. In some embodiments, the detector calibrationfilter 318 can be configured to automatically determine some or all ofthe parameters needed to calibrate a detector 108. The parametersdetermined in calibration of the detectors can be stored in thecalibration database 316 for later use by the tracking engine 304.

The target calibration filter 320 can be configured to calibrate thesystem to one or more specific targets or markers. For example, thetarget calibration filter 320 can be configured to, upon initial startupof a motion tracking routine, analyze images received from the detectors108 to determine a region of interest in the images where it is mostlikely that the optical marker exists. This information can be stored inthe calibration database 316. This initial calibration upon startup of amotion tracking routine can, for example, help to speed up a trackingroutine.

The tracking engine 304 comprises a marker location filter 312 and anobjection orientation filter 314. The tracking engine 304 can beconfigured to track the location of one or more optical markers duringan imaging scan and to determine an estimate of the pose of the objector patient being scanned. In some embodiments, the marker locationfilter 312 can be configured to analyze images from the detectors 108 todetermine locations of reference points of an optical marker in the 2Dimages from the detectors. The marker location filter 312 can beconfigured to analyze these images to determine both the locations ofthese points and the orientation of the reference shape or shapes formedby the reference points. The object orientation filter 314 can beconfigured to utilize the marker location information from the markerlocation filter 312 to convert that information into an estimated objectpose. The object orientation filter 314 can be configured to then passthe estimated object pose information off to the motion database 310and/or the controller interface 308 for use by the scanner controller106.

The tracking engine 304 can utilize various processes or algorithms indetermining the location of marker reference points and converting thisinformation into estimated object poses. Some examples of theseprocesses or algorithms are described in more detail below. However,various other processes or algorithms can be used with the techniquesdisclosed herein.

The controller interface 308 can be configured to convert object poseinformation into a coordinate system of the imaging scanner and to passthis information to the scanner controller 106. The controller interface308 comprises a coordinate system converter 322 and a motion data feedgenerator 324. The coordinate system converter 322 can be configured totake the estimated object pose information from the tracking engine 304and/or the motion database 310 and convert that object pose informationfrom the motion tracking system's coordinate system into the scanner'scoordinate system. The motion data feed generator 324 can be configuredto take the converted object pose information and transmit it to thescanner controller 106. In some embodiments, the motion data feedgenerator 324 is configured to transmit object pose information to thescanner controller 106 immediately as the object pose informationbecomes available. In other embodiments, the motion data feed generator324 can be configured to sync the timing of the motion tracking system102 with the scanner controller 106. For example, the motion trackingsystem 102 may be configured to acquire images and estimate object posesat a rate of approximately 100 hertz. The scanner controller 106 andscanner 104 may, on the other hand, be configured to take scans at adifferent rate. The motion data feed generator 324 can therefore beconfigured to match the motion tracking system's output speed to thescanner and/or scanner controller speed. In some embodiments, this mayinvolve caching converted object pose information until it is necessaryto send that information to the scanner controller 106. In someembodiments, the motion data feed generator 324 is configured to obtainmultiple object poses and to combine these, such as by averaging, beforesending them to the scanner controller 106. In other embodiments, themotion data feed generator 324 is configured to transmit only the latestobject pose estimate to the scanner controller 106. In some embodiments,the motion data feed generator 324 is configured to retransmit the lastobject pose estimate sent to the scanner controller 106 if the motiontracking system 102 has not generated a new object pose estimate sincethe last time the motion data feed generator 324 sent an object poseestimate to the scanner controller 106.

As described above, although the motion compensation system 300illustrated in FIG. 3A is disclosed in a system configured to adjust ascanner in real time or substantially in real time to adjust for patientmotion, other embodiments may utilize a motion compensation system toadjust acquired images after the scanning process to remove motionartifacts. In some embodiments, the scanner can be configured to adjustfor motion in real time and to post process images to remove anyremaining motion artifacts based on the tracked motion.

Although the motion compensation system 300 illustrated in FIG. 3Aillustrates the motion tracking system 102 as being a separate systemfrom the scanner controller 106, in some embodiments, the motiontracking system 102 can be integrated into the scanner controller 106.For example, a medical imaging scanner can be produced with anintegrated scanner controller comprising features to both control thescanner and to track object or patient motion during scan.

FIG. 3B is a block diagram depicting another embodiment of a motioncompensation system 301. The motion compensation system 301 illustratedin FIG. 3B is similar to the motion compensation system 300 illustratedin FIG. 3A. The motion compensation system 301, however, utilizes adifferent tracking engine 334 than the tracking engine 304 illustratedin FIG. 3A. The tracking engine 334 comprises a reference point filter336, a principal quantities generator 338, a reference frame translator340, an error filter 342, and a convergence filter 344. The componentsof the tracking engine 334 can be configured to operate to track themotion of an object or patient using, for example, the process flowdescribed below with reference to FIG. 7C.

The reference point filter 336 can be configured to, among other things,analyze two dimensional images of an optical marker to determine thelocations of the reference points of that marker. The principalquantities generator 338 can be configured to analyze the referenceshape formed by the reference points, as viewed by two or moredetectors, to determine a number of principal quantities, such as six,that can be used to help describe or define the 3D position andorientation of the optical marker.

The reference frame translator 340 can be configured to convert betweenthe two dimensional reference frame of each detector and the threedimensional frame of the motion tracking system. The error filter 342can be configured to analyze differences in principal quantities basedon the visualized reference points and based on estimates of an objectpose to determine an amount of error between the two. The convergencefilter 344 can be configured to perform iterative processes to reduce anamount of error in an object pose estimate until an object pose estimatehas an acceptable amount of error. The tracking engine 334 can beconfigured to communicate with the motion database 310, calibrationengine 306, and controller interface 308 similarly to as described abovewith respect to the motion compensation system 300 of FIG. 3A.

Other Embodiments of Motion Compensation Systems

FIG. 4A is another embodiment of a schematic diagram illustrating afront view of a medical imagine scanner 104 as part of a motioncompensation system 400. FIG. 4B is a schematic diagram illustrating aside view of the medical imaging scanner 104 as part of the motioncompensation system 400. The motion compensation system 400 is similarto the motion compensation system 100 illustrated in FIGS. 1A and 1B.However, the motion compensation system 100, as described above,comprises two detectors 108 positioned at a 90 degree angle to eachother along a transverse axis of the scanner 104. In the motioncompensation system 400, the detectors 108 are positioned at a 90 degreeangle 422 to each other along the longitudinal axis of the scanner 104.The detectors 108 of the motion compensation system 400 are stillconfigured to view the optical marker 110 along two different lines ofsight 420. The motion compensation system 400 illustrates that thedetectors 108 can be positioned in various ways, as long as eachdetector 108 views the optical marker 110 along a different line ofsight. The angle 422, as described above with respect to the angle 122,can vary and be larger or smaller. In some embodiments, the angle 422can be between 100 degrees and 70 degrees. In other embodiments, theangle 422 can be 30 degrees. For example, FIG. 4K illustrates a motioncompensation system 490 similar to the motion compensation system 400,except that the angle 422 is 30 degrees. In other embodiments, the anglecan be various other angles, as long as the two lines of sight 420 aredifferent. Further, in some embodiments, as described above, a motioncompensation system may comprise more than two detectors, such as four,six, eight, and the like, to add redundancy to the motion compensationsystem. For example, an MRI head cage may obstruct a view of a markerwhen the patient is in a particular position. However, if there areredundant detectors, there is an increased chance that robust motiontracking will continue despite one or more detectors' lines of sightbeing obstructed. In some embodiments, the detectors may be positionedas is illustrated in FIG. 4K, but at different angles 122. For example,a first set of two detectors 108 may be positioned at an angle 122 of 30degrees, and a second set of redundant detectors 108 (for a total offour detectors) may be positioned at a different angle, such as, forexample, 40, 50, 60, 70, 80, or 90 degrees.

FIG. 4C is another embodiment of a schematic diagram illustrating afront view of a medical imaging scanner 104 as part of a motioncompensation system 430. FIG. 4D is a schematic diagram illustrating aside view of the medical imaging scanner 104 as a part of the motioncompensation system 430. The motion compensation system 430 is similarto the motion compensation system 100 illustrated in FIGS. 1A and 1B.However, the motion compensation system 430 further comprises a headcage or head coil 432 configured to be positioned around a patient'shead. In certain medical imaging tasks, such as certain MRI head scans,a head cage 432 can be utilized and positioned around the patient'shead. The head cage can make it more difficult for a detector 108 toimage the optical marker 110 if the detectors 108 were mounted to thebore of the scanner body 104. Accordingly, the motion compensationsystem 430 comprises two detectors 108 mounted to the head cage insteadof the scanner body. The detectors 108 and motion tracking system 102are configured to operate similarly to as described above. The term headcage as utilized herein may be used to describe a device configured tohelp position the head of a patient during an MRI scan. The term headcage may also refer to a head coil device configured to wrap around apatient's head to perform MRI scanning functions.

FIG. 4E is another embodiment of a schematic diagram illustrating afront view of a medical imaging scanner 104 as part of a motioncompensation system 440. FIG. 4F is a schematic diagram illustrating aside view of the medical imaging scanner 104 as a part of the motioncompensation system 440. The motion compensation system 440 is similarto the motion compensation system 430 illustrated in FIGS. 4C and 4D.However, in some cases, there can be limited space within the bore of ascanner 104. In those cases, it can be difficult to position detectors108 to have a direct line of sight between their lens and the opticalmarker 110. Accordingly, the motion compensation system 440 comprisestwo detectors 108 positioned flat against the head cage 432 with a lineof sight 120 being through a mirror 442 to the optical marker 110. Themirrors 442 enable an indirect line of sight to make the system morecompact but to still enable viewing of the optical marker 110 from alongtwo different lines of sight 120. Although this embodiment illustratesthe use of mirrors with detectors mounted to a head cage, various otherembodiments may use mirrors and/or detectors attached to the scannerbody, the head cage, or any other location, as long as the detectors canview the optical marker through the mirrors. In some embodiments,multiple mirrors are used to redirect the line of sight 120 multipletimes. For example, a detector 108 may be positioned outside of thescanner and have its line of sight pass through one or more mirrorspositioned within the scanner to image the optical marker.

Although the motion compensation system 440 comprises mirrors toredirect the lines of sight, other methods of redirecting a line ofsight may be used, alone or in combination with mirrors. For example,fiber optics or prisms may be used to redirect a line of sight andcreate a virtual scissor angle.

FIG. 4G is another embodiment of a schematic diagram illustrating afront view of a medical imaging scanner 104 as part of a motioncompensation system 450. The motion compensation system 450 is similarto the motion compensation system 100 illustrated in FIGS. 1A and 1B.However, the motion compensation system 450 comprises two opticalmarkers 110. In this embodiment, each of the two optical markers 110 ispositioned to be directly in the line of sight of one of the detectors108. However, in other embodiments, multiple optical markers 110 may beutilized and positioned in other ways. For example, multiple opticalmarkers may be positioned at various rigid or substantially rigidportions of the object being imaged. For example, as further describedbelow, one optical marker 110 may be positioned on a patient's topteeth, while one or more other markers may be positioned on a patient'sforehead.

Optical markers may also be positioned at locations that are not rigidor substantially rigid. For example, an optical marker may be attachedto a patient's skin. In some embodiments, such as when the marker isattached to a patient's skin, due to skin movement or skin elasticity,the marker may at times move in relation to the object being scanned,which can introduce inaccuracies into a medical imaging scan.Accordingly, in some embodiments, a motion compensation system can beconfigured to differentiate between movements of the object beingscanned, such as a patient's head, and skin movement, which may notcorrelate to actual movement of the object being scanned. In someembodiments, the system can be configured to compare the positioning oftwo or more markers relative to themselves in order to differentiatebetween head movement and skin movement.

Utilizing multiple optical markers 110 can have various benefits. Forexample, multiple markers may be used for redundancy, in case one ormore markers is not currently visible to one or more detectors based onthe current object's pose. Another advantage is that multiple opticalmarkers can be analyzed simultaneously by the motion tracking system 102to obtain multiple object pose estimates. Those multiple object poseestimates can then be combined to generate a single more accurateestimate. For example, the multiple estimates can be averaged to come upwith an average estimate. In another example, there may be a measure ofmargin of error for each estimate and the estimates may be combinedusing a weighted average based on the margin of error. In otherembodiments, only the most accurate estimate is used and other estimatesare dropped.

FIGS. 4H-4J illustrate additional embodiments of motion compensationsystems configured to use indirect lines of sight. Given that manymedical imaging systems have limited space within the bore of thedevice, it can be advantageous to position detectors to be generallyflat against a bore of the device or flush within the bore of thedevice. The embodiment of a motion tracking system 460 shown in FIG. 4Hillustrates a system wherein two optical detectors 108 are positionedflat against a bore of the medical imaging scanner 104. In thisembodiment, the detectors 108 are positioned facing each other along alongitudinal axis of the bore. Two mirrors 442 are positioned relativelyclose to the detectors to redirect their lines of sight 120 toward theoptical marker 110. In this embodiment, the scissor angle issignificantly smaller than 90 degrees. However, in other embodiments,the detectors and/or mirrors may be positioned differently to increaseor decrease the scissor angle.

The motion compensation system 470 illustrated in FIG. 4I is similar tothe motion compensation system 460 illustrated in FIG. 4H. However, themotion compensation system 470 comprises two detectors 108 and twomirrors 442 mounted within the medical imaging scanner 104 such thatthey do not protrude into the bore of the scanner 104. The scanner 104body can comprise openings to enable the lines of sight 120 to pass fromthe marker 110 to the detectors 108. In some embodiments, detectors maybe positioned on a surface of the scanner bore, partially within thebody of the scanner, fully within the body of the scanner, and/or thelike. One determining factor of whether detectors can be mounted withina scanner body and/or whether any of the detector must protrude beyondthe scanner body is the size of the detectors and the space availablewithin the scanner body. More space available within the scanner bodyand/or smaller detectors may enable more or all of the detectors to bepositioned within the scanner body.

FIG. 4J illustrates a motion compensation system 480. The motioncompensation system 480 is similar to the motion compensation system 460illustrated in FIG. 4H. However, the motion compensation system 480comprises a head cage 432, and the detectors 108 and mirrors 442 aremounted opposite each other on opposite ends of the head cage 432,rather than being mounted to the bore of the scanner. In variousembodiments, the detectors 108 may be mounted in various positions, notnecessarily facing each other. For example, both detectors 108 may bepositioned on the same side of the head cage 432. As can be seen in FIG.4J, each of the two detectors 108 is configured to view the opticalmarker 110 along a line of sight 120 viewing the optical marker 110along a different angle relative to the marker. The line of sight 120 onthe left-hand side is at a shallower angle than the line of sight 120 onthe right-hand side. In other embodiments, the positioning of thedetectors, the optical marker, and/or the mirrors may be adjusted toadjust the angles of each of the lines of sight relative to the markerand/or to adjust the scissor angle.

FIGS. 4L-4N illustrate additional embodiments of schematic diagrams ofmotion compensation systems. Each of the motion compensation systemsillustrated in FIGS. 4L-4N comprise one or more detectors 108, multiplelines of sight 120, one or more optical markers 110, and a head cage orhead coil 432. Some embodiments also comprise one or more mirrors 442and one or more beam splitters 444. In some embodiments, the beamsplitters 444 comprise prisms. Although each of the motion compensationsystems illustrated in FIGS. 4L-4N include two optical markers 110, theconcepts disclosed herein may be applied to a system using any number ofoptical markers. In some embodiments, the motion compensation systemuses two optical markers to add redundancy to the system. For example,one optical marker may at some times have lines of sight to it blockedby, for example, a portion of the head coil 432. The optical markers canbe positioned such that when one optical marker's line of sight isobstructed, the other optical marker is still able to be imaged by oneor more detectors 108. Although the optical markers 110 illustrated inFIGS. 4L-4N comprise a checkerboard pattern, the optical markers 110 mayalso or alternatively comprise a pattern similar to the patternillustrated in the optical marker 110 shown in FIG. 2A.

The embodiment illustrated in FIG. 4L comprises four detectors 108imaging two optical markers 110. Each of the optical markers 110 issimultaneously imaged by two of the optical detectors 108 along twodifferent lines of sight 120. The embodiment illustrated in FIG. 4M issimilar to the embodiment illustrated in FIG. 4L, except the number ofdetectors is reduced by two. In this embodiment, each of the twodetectors 108 is configured to simultaneously image both of the opticalmarkers 110. The system is configured to accomplish this by splitting asight line from a detector through a beam splitter 444 and alsoredirecting the resulting multiple sight lines through mirrors 442. Aresulting digital image from the optical detector 108 may comprisedigital representations of both optical markers 110 (or potentially onlyone optical marker if, for example, one of the lines of sight isblocked). The motion compensation system can be configured to analyzethe representations of each of the optical markers separately, even ifthe representations are included in the same digital image.

The embodiment illustrated in FIG. 4N is similar to the embodimentillustrated in FIG. 4M, except only one optical detector 108 isutilized. The sight line of the optical detector 108 is split at beamsplitter 444 into two lines of sight 120, with each line of sight beingdirected toward one of the two optical markers 110. As with theconfiguration illustrated in FIG. 4M, a digital image resulting from thedetector 108 may comprise representations of both optical markers 110(or potentially only one optical marker if, for example, one of thelines of sight is blocked).

Mounting of Optical Markers

Optical markers can be mounted to an object being tracked in variousways. FIGS. 5A-5E illustrate various options for mounting an opticalmarker to an object. FIG. 5A is a front perspective view of anembodiment of an optical marker connected to a mounting portion 514.FIG. 5B is a side perspective view of the optical marker 110 of FIG. 5Amounted to the mounting portion 514. The optical marker 110 illustratedin FIGS. 5A and 5B is similar to the optical marker 110 illustrated inFIGS. 2A-2C. This embodiment is configured to attach to a patient's topteeth. The mounting portion 514 comprises a front portion 516, a backportion 518, and a trough 520 between the front portion 516 and backportion 518. The front portion 516 is configured to pass in front of aperson's top teeth, while the back portion 518 is configured to passbehind a user's top front teeth. The trough 520 is configured to conformor substantially conform to the user's teeth.

In some embodiments, the mounting portion 514 is configured to be shapedto a specific user to enhance the fit and/or rigidity of the opticalmarker 110. For example, the mounting portion 514 can be configured tobe softened and placed onto the user's teeth and then hardened. In oneembodiment, the mounting portion 514 comprises a polymer that softenswhen heated and hardens when cooled. In that example, the mountingportion 514 may, for example, be heated in warm water and then placed inthe user's mouth to set. In another embodiment, the mounting portion 514comprises an ultraviolet curable polymer. For example, the mountingportion 514 can be placed on a user's teeth in a soft condition and thencured to a hardened condition using an ultraviolet light. In otherembodiments, the mounting portion 514 can be configured to have anadhesive or moldable polymer material placed into the trough 520 toconform to or adhere to a user's teeth.

The mounting portion 514 can be connected to the optical marker 110using the connection portion 522. In some embodiments, the opticalmarker 110 and mounting portion 514 are an integral inseparable unit. Inother embodiments, the connection portion 522 enables the optical marker110 to be detached and reattached to the mounting portion 514. This maybe advantageous, for example, to enable the mounting portion 514 to beshaped to a user's teeth without the optical marker 110 mounted to it.In some embodiments, the connection portion 522 is relatively small andenables the optical marker 110 to be relatively close to the mountingportion 514. In other embodiments, the connecting portion 522 is longerand enables the optical marker 110 to be positioned further away fromthe object being tracked. In some embodiments, the connection portion522 is configured to position the optical marker 110 such that it willnot be covered by a patient's lips when the mounting portion 514 ispositioned over the user's teeth. In some embodiments, the mountingportion 514 comprises an additional flap configured to be positionedover a user's lip to keep their lip from blocking the optical marker110.

In some embodiments, an optical marker can be internally lighted. Forexample, an optical marker can include one or more LED lights or thelike inside the optical marker to uniformly or substantially uniformlylight the optical marker or individual reference points. Illumination ofan optical marker can be advantageous to enable a detector to moreclearly image the optical marker and to make it easier for a motiontracking system to distinguish the reference points on the opticalmarker and/or to distinguish centroids of the reference points. In someembodiments, the detectors and/or another part of a motion compensationsystem includes lights configured to illuminate the optical marker. Insome embodiments, such as when infrared detectors are used, the lightingcan be infrared lights. In some embodiments, an optical marker can beconfigured to glow in the dark to enable enhanced illumination withoutusing auxiliary lighting. In some embodiments, an optical marker can beconfigured to be charged with light by being placed in front of a lightsource for a period of time, after which the optical marker will glowfor a period of time.

FIG. 5C is a top view of another embodiment of an optical marker 110connected to a mounting portion 524. FIG. 5D is a top view of yetanother embodiment of an optical marker 110 connected to a mountingportion 534. The mounting portion 524 illustrated in FIG. 5C is arelatively narrow mounting portion comprising a front portion 526, backportion 528, and a trough 530 therebetween. The mounting portion 534illustrated in FIG. 5D is, however, a relatively wide mounting portion.The mounting portion 534 comprises a front portion 536, a back portion538, and a trough 540 therebetween. The embodiments illustrated in FIGS.5C and 5D illustrate that mounting portions can take various shapes. Forexample, the mounting portion 524 is similar in size to the mountingportion illustrated in FIGS. 5A and 5B. This mounting portion may engageonly a couple teeth of the patient. On the other hand, the mountingportion 534 illustrated in FIG. 5D comprises a more U-shaped mountingportion configured to engage several teeth of the patient. The mountingportion 534 may be in some embodiments similar in design to a mouthguardtypically used by athletes. The mounting portions 524 and 534 are bothconnected to an optical marker 110 using a connection portion 532.

FIG. 5E is a perspective view of an embodiment of an optical marker 110connected to a patient 112. In this embodiment, the optical marker 110is attached to a strap 550 configured to go around the patient's head tohelp retain the optical marker 110 in position. In some embodiments, theoptical marker 110 also is attached to a mounting portion, such as themounting portions described above configured to attach to a patient'steeth. Various embodiments of mounting an optical marker to a patienthave been described. However, various other methods may be used such as,but not limited to, adhesives, tape, tattoos, paint, and/or the like.

FIG. 6A is a perspective view of an embodiment of an optical marker 610using distributed reference points 614 and 612. Like the optical marker110 illustrated in FIG. 2B, the optical marker 610 comprises threereference points 612 and 614 comprising elliptical shapes and forming atriangular reference shape when viewed together. However, the opticalmarker 610 is a distributed optical marker, in that the reference pointsare not part of the same physical object. In this embodiment, thereference point 612 is on a substrate attached to the patient's topteeth, similar to as described above. However, the other two referencepoints 614 are stickers attached to the patient's forehead above his orher eyes. One advantage of the configuration illustrated in FIG. 6A isthat detectors that have a lower resolution, and are therefore cheaper,may be used. Another advantage is that smaller movements of the patient112 may be able to be more accurately detected in images of the opticalmarker 610. A disadvantage of the configuration shown in FIG. 6A is,however, that the three reference points are not rigid with respect toeach other. For example, if the patient 112 moves his or her skin overthe forehead, the reference points 614 may move with respect to thereference point 612 attached to his or her teeth. In some embodiments,the motion tracking system can be configured to analyze relative motionof the reference points to determine whether the motion is due to localskin motion or deformations or actual patient movement.

The concept of utilizing distributed reference points may take variousforms. For example, each reference point may be positioned on a block,substrate, or other backing, with the backing being affixed or attachedto the patient or object being tracked. For example, the reference point612 illustrated in FIG. 6A is shown attached to a substantially rigidbacking. The reference points 614 are illustrated as individual pointsnot having a backing and being directly affixed to the patient'sforehead. However, in some embodiments, the reference points 614 mayalso be attached to a backing, block, or other backing to help inattaching the reference point to the object and/or to reduce any flexingof the reference point and/or to keep the reference point shape flat. Insome embodiments, an optical marker and/or an individual reference pointcan be configured to attach to an object being tracked using a varietyof means. For example, double-sided surgical tape can be utilized toattach a marker to an object. Other adhesives may also be used. Othermechanical affixing means may also be used, such as other types of tape,clamps, straps, fasteners, and/or the like.

Although the embodiments illustrated in FIG. 6A illustrate referencepoints located at a patient's top teeth and forehead, reference pointsand/or markers may be positioned at various locations. For example, areference point or marker may be positioned at a patient's nose, cheeks,chin, temples, ears, and various other locations as long as thereference point or marker will be visible to a detector during at leasta portion of the object's range of motion, either through a direct orindirect line of sight. Further, although the reference pointsillustrated in FIG. 6A are spaced relatively far apart from each other,in some embodiments, the reference points may be spaced relatively closetogether, although still not attached to the same backing or mountingmaterial. For example, the reference points may be located and spaced asimilar distance apart as shown in the optical marker 110 illustrated inFIG. 2B, even if the three reference points are each attached to adifferent backing material or mounting portion. In another embodiment,the three reference points can be spaced in a similar fashion to theoptical marker 110 illustrated in FIG. 2B, except the reference pointsare each a separate object adhered to the patient and not connected toeach other.

In some embodiments of a motion compensation system, obstructions mayexist that could potentially block a line of sight to an optical marker.For example, a head cage or head coil, such as is shown in FIG. 9A mayhave bars or other portions that may obstruct a line of sight to amarker. Accordingly, in some embodiments, a marker or reference pointcan be positioned on an object at a location that eliminates or reducesany blockage of a line of sight to the marker. In some embodiments,mirrors or other devices configured to redirect a line of sight may beused to enable a detector to image a marker through a hole or opening inan obstruction such as a head cage or head coil.

FIG. 6B is a perspective view of an embodiment of an optical marker 610using distributed reference points 614 and 612, and an embodiment of anoptical marker 110 using integrated reference points 612 and 616. Theembodiment illustrated in FIG. 6B is similar to the embodimentillustrated in FIG. 6A, but utilizing two different-sized opticalmarkers. The configuration in FIG. 6B utilizes the distributed opticalmarker 610 illustrated in FIG. 6A and the non-distributed optical marker110 illustrated in FIG. 2B. Such a configuration can be advantageous to,for example, obtain the benefits of a larger marker 610 whilecounteracting some of the issues with a distributed marker by using thenon-distributed marker 110.

In some embodiments, a motion tracking system can be configured to trackthe larger marker 610 and/or the smaller marker 110 based on the motionthe detectors are capturing. For example, if a system is concerned withlocal motion or twitching of the patient's forehead skin that mayintroduce inaccuracies into the system, the system may be configured tostop tracking utilizing the larger optical marker 610 when motion in adirection consistent with a localized motion of a marker 614 isdetected. For example, if a marker 614 is imaged moving in alongitudinal direction, such as up and down with respect to thepatient's head, this may be indicative of local skin motion and mayintroduce inaccuracies if tracking the optical marker 610 is used.However, if motion of the reference point 614 in a side-to-sidedirection is detected, especially if both reference points 614 aremoving in the same direction, this is less likely indicative of localskin motion and is more likely that the patient is turning his or herhead. Accordingly, tracking based on the larger optical marker 610 maybe more accurate during that motion than tracking the smaller opticalmarker 110. In one embodiment, a motion tracking system can beconfigured to track motion utilizing a larger distributed marker whenmotions not indicative of localized skin motion are detected, but totrack using a smaller non-distributed marker when motions indicative oflocalized skin motion are detected.

FIG. 6C is a perspective view of an embodiment of multiple opticalmarkers 110 attached to an object being tracked, in this case thepatient 112. This embodiment illustrates that multiple complete opticalmarkers 110 can be affixed to the same object. Although, in thisembodiment, the optical markers are attached at the patient's top teethand forehead, optical markers may be attached in addition to or in lieuof these locations at, for example, the cheeks, the temples, ears, chin,nose, below the eyes, and/or the like. Utilizing multiple opticalmarkers may be advantageous to increase accuracy and/or redundancy.

Motion Compensation System Processes

FIG. 7A depicts an embodiment of a process flow diagram illustrating anexample of a motion compensation process. For example, the process flowillustrated in FIG. 7A can be implemented by, for example, the motioncompensation systems 300 or 301 illustrated in FIGS. 3A and 3B. At block702, the scanner starts an imaging scan. At block 704, the motioncompensation system starts its process. At block 706, a tracking engineacquires marker images. For example, the tracking engine 304 illustratedin FIG. 3A can be configured to acquire one image frame each fromdetector 1 and detector 2. At block 708, the tracking engine determinesan object pose. For example, the marker location filter 312 can beconfigured to determine two dimensional locations of marker referencepoints, and the object orientation filter 314 can be configured toestimate a three dimensional pose of the object being tracked using thetwo dimensional locations of the marker reference points. The trackingengine may store this information in the motion database shown at block710. The motion database shown at block 710 may be, for example, themotion database 310 illustrated in FIG. 3A.

At block 712, a controller interface generates motion data. For example,the controller interface 308 illustrated in FIG. 3A can be configured toconvert the object pose information into the scanner coordinate systemand generate data for enabling the scanner controller to adjust for themotion. At block 714, the controller interface transmits the motion datato a scanner controller. At block 716, the scanner controller operatesto adjust the scanner to compensate for the motion. At block 718, theprocess varies depending on whether the imaging scan is complete. If theimaging scan is not complete, the process flow proceeds back to block706 and proceeds as described above. If the scan is complete, theprocess flow proceeds to block 720, and the motion compensation processis completed.

FIG. 7B depicts another embodiment of a process flow diagramillustrating an example of a motion compensation process. At block 722,a scanner starts the imaging process. At block 724, a motioncompensation system starts its process. For example, this process may beimplemented by the motion compensation system 300 illustrated in FIG.3A. At block 726, the motion tracking system is optionally calibrated.For example, the target calibration filter 320 can be configured todetermine a general location of the optical marker in each detector'sfield of view.

At block 728, a tracking engine acquires simultaneous marker images fromtwo different vantage points. For example, the tracking engine 304illustrated in FIG. 3A can be configured to obtain simultaneous orsubstantially simultaneous images from the first and second detectors108. At block 730, a marker location filter analyzes the images todetermine two dimensional positions of reference points of an opticalmarker. For example, the marker location filter 312 can be configured toanalyze the images acquired in block 728. The marker location filter canbe configured to analyze these images to determine where the referencepoints of the optical marker appear in each image.

At block 732, an object orientation filter analyzes the reference pointpositions to determine a three dimensional object orientation or pose.For example, the object orientation filter 314 can be configured toanalyze the information created at block 730 to estimate a threedimensional object pose in 6 degrees of freedom and, in someembodiments, to perform an iterative process to obtain a betterestimate. The objection orientation filter can be configured to storethis 3D object orientation or pose information in the motion databaseillustrated at block 734. At block 736, the controller interface can beconfigured to generate motion data. For example, the controllerinterface 308 illustrated in FIG. 3A can be configured to acquire theobject pose or orientation information from the motion database or theobject orientation filter and convert this information to be readable bya scanner controller. At block 738, the controller interface transmitsthe motion data to the scanner controller. At block 740, the scannercontroller adjusts the scanner to compensate for the motion.

At block 742, the process flow varies depending on whether the imagingscan is complete. If the imaging scan is not complete, the process flowproceeds back to bock 728 and proceeds as described above. If theimaging scan is complete, the process flow proceeds to block 744 and themotion compensation process is completed.

FIG. 7C depicts another embodiment of a process flow diagramillustrating an example of a motion compensation process. The processflow illustrated in FIG. 7C can be performed by, for example, the motioncompensation system 301 illustrated in FIG. 3B. At block 752, a scannerimaging scan is started. At block 754, the motion compensation systemstarts its process. At block 756, the motion compensation systemoptionally calibrates the tracking system. At block 758, a trackingengine acquires two simultaneous marker images from different vantagepoints. For example, the tracking engine 334 can be configured toacquire simultaneous or substantially simultaneous image frames from thefirst and second detectors 108.

At block 760, a reference point filter locates reference points and thefirst two dimensional image. For example, the reference point filter 336can be configured to analyze the first two dimensional image to locatethe alternating ellipses or circles of the optical marker illustrated inFIG. 2A and to determine a centroid of each of those sets of alternatingellipses. The centroid will be the reference point. At block 762, thereference point filter locates the reference points in the seconddimensional image. Accordingly, after blocks 760 and 762, if the opticalmarker illustrated in FIG. 2A is being used, six different coordinatesfor six different reference points will be or will have been determined.The six different reference points comprise the three reference pointsof the optical marker as seen by the first detector and the same threereference points as seen by the second detector.

At block 764, a principal quantities generator calculates baselineprincipal quantities or attributes based on the two dimensionalreference point locations. For example, the principal quantitiesgenerator 338 illustrated in FIG. 3B can be configured to analyze thelocations of the reference points as determined in blocks 760 and 762 todetermine six principal quantities describing various attributes of thereference shape formed by the reference points and/or describingdifferences or similarities between the reference shape as viewed alonga first and second line of sight. These baseline principal quantitiescan be utilized at another point in this process to be compared toprincipal quantities based on an estimate of an object pose to determinean amount of error. Further discussion on the principal quantities andspecific examples of principal quantities can be found below.

At block 766, a reference frame translator accesses an estimated objectpose. For example, the reference frame translator 340 illustrated inFIG. 3B can be configured to access a motion database shown at block 768to retrieve an estimated object pose. In some embodiments, the estimatedobject pose is arbitrary, such as (0, 0, 0, 0, 0, 0) in a 6 degree offreedom orientation. In other embodiments, the estimated object poseretrieved at block 766 is the immediately prior estimated object posefrom the last time this process was performed.

At block 770, a reference frame translator calculates expected locationsof reference points based on the estimated object pose. For example, thereference frame translator 340 illustrated in FIG. 3B can be configuredto convert the three dimensional estimated object pose from block 766into expected three dimensional and two dimensional locations of thereference points as if the object were in that estimated pose. At block772, the principal quantities generator calculates comparison principalquantities based on the expected reference point locations. Theprincipal quantities generator can be configured to calculate theseprincipal quantities similarly to how it calculated the baselineprincipal quantities at block 764; but at block 772 the principalquantities are based on the expected reference points as a result of theestimated object pose.

At block 774, an error filter determines an amount of error between thebaseline and comparison principal quantities. For example, the errorfilter 342 illustrated in FIG. 3B can be configured to compare thebaseline and comparison principal quantities and determine aquantitative error estimate. At block 776, the process flow variesdepending on whether the error determined at block 774 is within anacceptable range. If the error is not within an acceptable range, theprocess flow proceeds to block 778. At block 778, a convergence filtercalculates an estimated object pose intended to reduce the error. Forexample, the convergence filter 344 can be configured to analyze theestimated object pose from block 766 and determine how to alter thatestimated pose to reduce the error. The process flow then proceeds backto block 770 and proceeds as described above using the new estimatedobject pose.

Returning to block 776, if the error is within an acceptable range, theprocess flow proceeds to block 780. At block 780, the tracking enginestores the estimated object pose in the motion database. At block 782, acontroller interface generates motion data. For example, the controllerinterface 308 illustrated in FIG. 32B can be configured to convert acoordinate system of the object pose into the scanner coordinate systemand to package this information for transmission to the scannercontroller. At block 784, the controller interface transmits the motiondata to the scanner controller. At block 786, the scanner controlleradjusts the scanner to compensate for the motion.

At block 788, the process flow varies depending on whether the imaginescan is complete. If the imaging scan is complete, the process flowproceeds to block 790 and the motion compensation process is completed.If the imaging scan is not complete at block 788, the process flowproceeds back to block 758 and proceeds as described above.

Although various embodiments described herein describe images frommultiple detectors being acquired simultaneously or substantiallysimultaneously, in some embodiments, the images must merely be acquiredwithin a certain time of each other. For example, in some embodiments,the images must just be acquired within a timeframe that is equal to orless than the frequency of the motion tracking system providing motionupdates to the scanner controller. Another example of a process flowdiagram illustrating an example of a motion compensation process isshown in FIG. 20C.

In some embodiments, the processes illustrated in FIG. 7C and asdescribed elsewhere in this disclosure requires a computer system and/orcomputer hardware to implement. For example, to enable real-time motiontracking during a medical imaging scan, in some embodiments, a motioncompensation system must operate at a speed of at least 100 Hz, forexample, performing an object orientation estimate at a rate of at least100 times per second. Such a process is impossible for a human toperform in his or her mind. Even if a human were able to implement theprocesses described herein, a human would not be able to implement themat the speeds necessary for some embodiments. However, even if speedwere not a requirement, a human would likely not be able to perform atleast some of the components of processes described herein in his or herhead due to the large number of variables and relatively complexequations involved. Further, the human eye likely cannot distinguishsmall movements of an optical marker within the resolution necessary toimplement some of the processes disclosed herein. Accordingly, computersystems and/or computer hardware is required to implement someembodiments of the processes and systems and methods described herein.

FIG. 7D depicts another embodiment of a process flow diagramillustrating an example of a motion compensation process. The processflow illustrated in FIG. 7D illustrates one example of a process ofusing multiple optical markers. In this embodiment, a motion trackingsystem is tracking one larger optical marker and one smaller opticalmarker, such as is shown in FIG. 6B. However, the concepts describedherein may be used in various other embodiments of multiple markersystems. At block 701, a scanner imaging scan is started. At block 703,a motion compensation system starts its motion compensation process. Atblock 705, the motion tracking system is optionally calibrated. At block707, a tracking engine acquires simultaneous marker images from twodifferent vantage points.

At block 709, a marker location filter analyzes images to determine twodimensional positions of reference points for the larger referenceshape. For example, a marker location filter can be configured todetermine the reference point locations of the reference points 614 and612 illustrated in FIG. 6B. At block 711, an objection orientationfilter analyzes the larger reference shape position to estimate a 3Dobject orientation or pose.

At block 713, the marker location filter analyzes the images from thedetectors to determine 2D positions of reference points for the smallerreference shape. For example, the marker location filter can beconfigured to determine the positions of the reference points 612 and616 illustrated in FIG. 6B. The locations of these reference pointsdefine the position of the reference shape defined by those referencepoints. At block 715, the object orientation filter analyzes the smallerreference shape position to estimate the 3D object orientation or pose.

At block 717, the tracking engine combines both object orientation orpose estimates from blocks 711 and 715 to produce a single estimate. Forexample, the tracking engine may average the estimates. In anotherexample, the tracking engine may use one estimate or the other dependingon which is most likely to be the more accurate estimate at this pointin time. The tracking engine can be configured to communicate with themotion database illustrated at block 719 to store the estimate. At block721, a controller interface generates motion data. For example, thecontroller interface can be configured to convert the object orientationor pose estimate into a scanner's coordinate system. At block 723, thecontroller interface transmits the motion data to a scanner controller.At block 725, the scanner controller adjusts the scanner to compensatefor the motion. At block 727, the process flow varies depending onwhether the imaging scan is complete. If the imaging scan is notcomplete, the process flow proceeds back to block 707 and proceeds asdescribed above. If the imaging scan is complete at block 727, theprocess flow proceeds to block 729 and the motion compensation processis completed.

Optical Target Fixed to an Anatomical Location, e.g., the Head

A challenge in optical head tracking is locating a head feature whichmoves rigidly with the body's skeletal frame. The skin is elastic andallows significant motion (relative to the displacement desiredaccuracy; for instance while blinking, twitching or wrinkling the noseor forehead. To overcome this challenge, in an embodiment, the system isconfigured to employ two or more optical tracking targets for placementon the patient. For example, two or more optical tracking targets can becoupled (for example, painted or affixed to the face of the patient) tothe skin of the patient. By employing two or more optical trackingtargets, the system can be configured to compensate for the elasticnature of the skin in order to determine the motion of the patient. Forexample, the system can be configured to track motion of the two or moreoptical tracking targets and average the detected motion in order todetermine the approximate motion of the patient. Alternatively, in anembodiment, the system can be configured to analyze the detected motionfrom the two or more optical tracking targets and compare the detectedmotion from each to a predicted motion value. The system can beconfigured to select the detected motion value that is closest to thepredicted motion value and ignore the rest of the detected values.Alternatively, the system can be configured to ignore the detectedvalues that are substantially different from the predicted motion value.The system can be configured to average the detected motion values thathave not been ignored. In an embodiment, the system can be configured toapply or combine one or more of the foregoing techniques.

To overcome the challenge of the elastic nature of skin, in anembodiment, an optical tracking target can be coupled to the upper teethof the patient. One accessible feature which is rigid to the skull isthe upper teeth. Unlike the teeth on the lower jawbone, the upper teethare rigidly affixed to the skull of the patient. In an embodiment, acompact and reliable optical tracking target can be attached to one ormore of the upper teeth with a clip-on or other coupling device. Suchattachment devices can be configured to be extremely comfortable. In anembodiment, a printed precision optical target is attached to the topfront teeth of a patient.

An optical target can be configured to be easy to locate with a highdegree of accuracy regardless of orientation in a sensor field of view.A circle or series of concentric circles or ellipses can be potentiallyadvantageous in this regard. Furthermore, to accommodate the fastestcomposite 2D data processing methods, a number (at least 3) of centroidpositions can be discernible at every instant in time. The target canbe, in some embodiments, composed of three sets of concentric circles orellipses located at the vertices of an equilateral triangle. Compactnessis desired for practical reasons, but the minimum size and spacing ofthe targets is dictated to large extent by characteristics of thesensors and the available non-occluded optical lines of sight throughthe MRI field compensation coil. A tradeoff arises, for instance,between the minimum size of the target and the cost of the imagingcameras used to sense the head motion—the smaller the edge dimension ofthe target triangle, the more pixels required in the camera sensor, andthe faster the readout and processing electronics required.

As a reasonable compromise, in some embodiments an equilateral triangleside length of 0.5 inches can be adopted. The printed target patternincludes a solid central elliptical dot of 1/16″ minor diameter at eachtriangle vertex, and each dot is surrounded by a first concentricellipse of 3/16″ minor diameter and 1/32″ line width, and a secondconcentric ellipse of 5/16″ minor diameter and 1/32″ line width(ellipses scaled to look circular from camera nominal 45° look angle).In this embodiment, the entire target measures about 1 inch wide byabout 0.93 inches high. Other dimensions are possible.

A camera viewing this target is able to determine the centroid of eachellipse on the target pattern using a simple brightness momentcalculation, independent of orientation of the target. The target itselfsubtends only a small portion of the camera field of view, but isrecognizable by its high contrast and lack of gray scale. In embodimentsthe computer processor is programmed to track each of the threesub-targets by enclosing each of the three sub-targets within asub-pixel array of 48×48 pixels and to calculate centroids of eachsub-target by dividing (a) the sum of the product of pixel darkness andpixel position by (b) the sum of the pixel darkness of all of the pixelsin the 48×48 sub-pixel array. The processor is also programmed to moveeach of the 48×48 pixel arrays so that its target is always locatedfully within the sub-pixel array. With sufficient camera spatial andbrightness resolution and target illumination and contrast, centroidpositional accuracy of about 0.1 pixels in row and/or column or less isachievable using this target.

FIG. 2A is an enlarged view of the optical target and two of the threeCartesian axes. FIG. 2C shows a full-scale target (compared to a U.S.penny) affixed to NTI coupling device for placement on the upper teethof the patient. (Lower right) Subject with optical target and nightguard clipped onto front teeth.

Latency

Latency in the measurement of head motion using optical trackingtechniques is comprised of the camera sensor integration and readouttime, the target centroid determination time and the 6-DOF decompositiontime. In order to reliably track head motions as fast as 2 cm/second andhead rotations as fast as 10 degrees per second, a camera frame rate ofabout 100 Hz is desired, with electronic shuttering to freeze motion atrates up to 10 times this speed for sharp resolution of the opticaltarget without blurring. A significant field of view is required toaccommodate large motions, so fast camera readout without expensivemechanical tracking capabilities will require either a low pixel densityor a camera with a larger focal plane but the ability to window asmaller region of interest for readout. Centroid and 6-DOF decompositionalgorithms running in composite 2D, rather than full 3D space, andutilizing rapidly converging solution methods can be capable ofreturning solutions to the compensating head coil electronics at 100solutions per second, with about 10 ms of latency. In some embodiments,the system can be configured to operate with a latency that enables itto update the scanner in between each image acquisition

Cameras

For a subject wearing or coupled with the optical head tracking target,the target size and subject rotation angles and translation positiondetermine the physical location of the three target centroids preciselyin three dimensions. With precise knowledge of these angles and theoptical sensor (camera and lens) parameters—pixel pitch, lens focallength and radial distortion, camera location and orientation relativeto nominal target position—the location of the target centroidprojections on the focal plane sensor can be predicted to any level ofaccuracy even prior to measurement.

In principle, the inverse problem should be equally simple as long asthe 3D position of the target centroids can be ascertained optically.Using two cameras, a stereo view of the centroid projections can be usedto determine the 3D location in space of each target centroid, and the6-DOF displacement vector can then be determined through a simple matrixinversion. In practice, however, this approach leads to expensive andcomplicated requirements on camera pixel density, pixel count, cameraalignment and camera calibration.

An alternate unfolding approach dispenses with stereo ranging but usesseparate 2D projections from two cameras without attempting to correlateabsolute target positions on the two cameras. This approach eliminatesthe strict requirements on camera alignment and magnification matchingcharacteristic of the stereo vision approach, and also relaxes the pixeldensity and count requirements needed to obtain the required positionalaccuracy (about 0.1 mm in translation and about 0.1 degrees in rotation)by about a factor of 20, resulting in significant savings in cost andprocessing speed. Although various embodiments described herein may notutilize stereo ranging, imaging, or vision, some embodiments may utilizeconcepts of stereo ranging, imaging, or vision, either alone or incombination with other object orientation determination techniquesdescribed herein.

Even for this 2D measurement approach some basic steps can be taken tocalibrate camera parameters once the cameras are integrated with thehead coil; these can be performed at the manufacturing facility. Theseinclude measuring the projected pixel location of a single referencepoint on both cameras, as well as the camera magnification factors forpixel displacement per degree of rotation in pitch, yaw and roll, andper mm of translation along x, y and z. However, as stated before, it isnot necessary that the cameras be exactly aligned in space (e.g.perfectly normal) or that their magnifications (lens focal length anddistance to reference point) be identical, as is easily verified bysimulation.

Stereo Versus Composite 2D Vision Requirements

With a single camera viewing the target from 45 degrees off of verticalin the target plane, the camera sees very little centroid displacementwhen the target moves in the direction of the camera (e.g. upwardvertical translation equal to horizontal translation in the cameradirection, with no rotation). Assuming a 7 micron pixel pitch, a 25 mmlens, and a working distance of 14 inches, target displacement in thecamera direction may be at least 0.6 mm before the target can bedetected as a 0.1-pixel increase in target centroid separation. However,as shown in FIGS. 8A and 8B a second camera placed orthogonally, e.g. at−45 degrees relative to vertical in the same plane, is maximallysensitive to this same motion, seeing a full pixel displacement of eachcentroid for a diagonal translation of only 0.1 mm. The second cameraeliminates the “blind spot” that a single camera has to motion along itsoptical axis. While certain embodiments described systems in whichcameras are positioned orthogonally, cameras can also be placed atrelative angles other than orthogonal with respect to vertical in thesame plane, depending on the desired clinical result.

Camera Depth of Field

To accommodate head roll of +/−15 degrees plus the 0.85-inch targetwidth at a working distance of 14 inches, the lens can be configured toprovide sharp focus for distances between 13″ and 15.5″. At f/22,assuming a circle of confusion slightly smaller than a camera pixel (7microns), a 25 mm focal-length lens provides this necessary depth offield a nominal 14-inch focus. At this working distance, the opticalpath can be folded with a turning mirror (FIG. 3) or otherwiseconfigured to fit within the 70 cm diameter bore of the main MRI coil. Anon-ferrous camera can be utilized in the MRI environment. In anembodiment, it can be cost effective to repackage a commercial camerafor use in the strong magnetic field.

In some embodiments, one possible camera that can be utilized ormodified for use with systems and methods as disclosed herein, isproduced by Allied Vision Technologies and designated the ProsilicaGE-680 Monochrome CCD Camera. This camera features a Kodak KAI-0340 ⅓″640×480 VGA focal plane sensor with 7.4 μm square pixels and a fastGigabit Ethernet output delivering up to 205 frames per second at 12-bitpixel depth. An inexpensive possible lens for use is an Edmund OpticsTechSpec 25 mm high-resolution fixed focal length lens.

For this camera and lens, at 14 inches from the target at 45° incidence,the 5/16″ diameter target circles project to ellipses on the camera,with the minor diameter of the largest ellipses at about 28 pixels andthe major diameter at about 40 pixels. With sufficient S/N ratio (targetillumination) and lens MTF (sharpness), this pattern should allowaccurate centroiding to about 0.1 pixels in row and/or column or less.The entire projected target subtends about 128 H×168 V pixels, andallowing for head roll of +/−11.5 degrees, a camera with 640 horizontalpixels (pixel columns) can accommodate the entire field of interestwithout mechanical tracking provisions.

FIGS. 9A and 9B show a modified head coil with cameras mountedlongitudinally and turning mirrors to accommodate a longer workingdistance than is possible with a straight optical path in theconstrained space of the main MRI coil bore. In embodiment, the systemis configured with two or more cameras with a direct view of the opticaltracking targets without the use of mirrors.

Six Degree-of-Freedom Measurement and Reporting Algorithm

In some embodiments, the MRI Head Tracker takes real-time input from two2D imaging sensors and analyzes these data to determine and reportmotions in six degrees of freedom with minimal latency. This task can beperformed by detecting and measuring the three centroid positions on thetarget and utilizing those positions with a reporting algorithm todetermine the position of the patient's head.

Six-Degree-of-Freedom Coordinate System

In an embodiment, the system is configured to use a coordinate systemfor reporting 6-DOF motions to the MRI field compensation system that isa Cartesian system aligned with the symmetry axis of the head coil asshown in FIGS. 10A and 10B. Head coil coordinate system shown in FIG.10A is coincident with body coordinates in the nominal (“square”) headposition as shown in FIG. 10B. The z direction is into and out of theplane of the drawing. Target displacements and rotations are reported tothe coil field compensation system using this system of coordinates.

Coordinate definitions are adopted by the same conventions used indefining aircraft motion, except that the rotation directions are takento be right-handed (positive for counter-clockwise rotation about thebasis direction vectors):

x is the longitudinal (chin-to-crown) direction, with values increasingtoward the top of the head

y is the transverse (left-to-right) direction, with increasing valuestoward the patient's right ear

z is the up-down direction, with increasing values toward the ceiling

$\psi = {\tan^{- 1}\left( \frac{\Delta \; y}{\Delta \; x} \right)}$

is the yaw angle or right-handed rotation about the z-axis (head leantoward shoulder while facing forward, zero at normal “square” position,positive values for patient leaning toward patient's right shoulder)

$\theta = {\tan^{- 1}\left( \frac{\Delta \; x}{\Delta \; z} \right)}$

is the pitch angle or right-handed rotation about the y-axis (nodding“yes,” zero at normal “square” position, positive values for patientlooking “upward”)

$\phi = {\tan^{- 1}\left( \frac{{- \Delta}\; y}{\Delta \; z} \right)}$

is the roll angle or right-handed rotation about the x-axis (shaking thehead “no,” zero at normal “square” position, positive values for patientlooking toward patient's left side).

The origin of coordinates and angle zero references are arbitrary, asonly relative motions are reported, however two convenient referenceorigin positions exist: 1) at the center of the target in its normal(“square”) head position, and 2) at the base of the neck at the pointwhere the head swivels for nod, turn and lean motions. The latter isadopted here (as shown in FIG. 2), simply for ease in orthogonalizingthe set of principal observation parameters with common motiondirections in the 6-DOF decomposition algorithm.

Target Displacement Equations

The full 6-DOF translation is composed of a 3-D displacement as well asa 3-axis rotation. To first order we assume that the skull moves as arigid body about a single rotation point somewhere in the neck. Fromthis point the translation becomes separable from the rotation, so thisis chosen as the coordinate origin. The rotations are separated intoroll, pitch and yaw as described above, and the translated positionthrough rotation follows the Euler rotation matrix formulation asfollows (using right-handed angle conventions). The x, y, and zdisplacement coordinates then follow the independent translations:

${{\begin{pmatrix}x^{\prime} \\y^{\prime} \\z^{\prime}\end{pmatrix} = {{\begin{pmatrix}1 & 0 & 0 \\0 & {\cos \; \phi} & {{- \sin}\; \phi} \\0 & {\sin \; \phi} & {\cos \; \phi}\end{pmatrix}\begin{pmatrix}{\cos \; \theta} & 0 & {\sin \; \theta} \\0 & 1 & 0 \\{{- \sin}\; \theta} & 0 & {\cos \; \theta}\end{pmatrix}\begin{pmatrix}{\cos \; \psi} & {{- \sin}\; \psi} & 0 \\{\sin \; \psi} & {\cos \; \psi} & 0 \\0 & 0 & 1\end{pmatrix}\begin{pmatrix}x \\y \\z\end{pmatrix}} +}}\quad}{\begin{pmatrix}{\Delta \; x} \\{\Delta \; y} \\{\Delta \; z}\end{pmatrix}.}$

Decomposing the six independent translations from the absolute andrelative displacements of the measured target centroids is the subjectof this effort. The 2D inverse problem is somewhat more difficult thanthe 3D problem, in that after the target centroid projections in focalplane row and column are determined, significant degeneracies remain inthe unfolding matrices for each camera. Combining the data from bothcameras removes these degeneracies through a series of interrelated,nonlinear equations. The fastest procedure for solving this inverseproblem is obtained by the Newton-Raphson method or a variant thereof,whereby an approximate first-order solution is proposed and testedagainst the known (measured) centroid locations on the two camera focalplanes. The residual error is divided by the local derivatives withrespect to each component of rotation and translation, to determine aniterative correction. The first-order solution is chosen by consideringthe features of the projected target pattern which are most stronglyaffected by a single rotation angle or displacement, and linearizing theinversion problem along these feature axes.

A 6-DOF motion simulation and decomposition algorithm was developed andtested to allow simulation of arbitrary motions and then verify theability of a pair of orthogonal cameras to decompose centroidmeasurements at the 0.1-pixel level into distinct x, y, z, roll, pitchand yaw components at the requisite level of accuracy.

Six-Degree-of-Freedom Motion Determination Algorithm

General subject motion is a superposition of translation along x, y, andz as well as rotation about the x, y and z axes (designated roll, pitchand yaw respectively). Displacements along each of these degrees offreedom are not sensitive to coordinate system origin; however it isconvenient (as explained above) for modeling purposes to place an originnear the region of the spine about which the head rotates and swivels,and a secondary reference point at the center of the optical trackingtarget in the nominal (“correct”) head position and orientation. Thissecondary reference is typically offset from the spinal origin by ˜10 cmin x and ˜10 cm in z.

The target shown in FIG. 1, as viewed from a single camera, appears asthree sets of concentric ellipses with centroids projected onto threedifferent positions (column, row) on the camera focal plane. The camerais centered along the (x=constant) plane of the target and aligned suchthat the central pixel row images this plane, at an angle of 45 degreeswith respect to both the y and z axes and with the nominal target centerprojected to the central pixel column. Using a camera with 7.4 micronpixels and a 25 mm lens, positioned at a distance of 14.1 inches fromthe nominal target center, centroids from the vertices of an equilateraltriangle target with sides of length 0.5 inches are projected onto thecamera focal plane as shown in FIG. 11. This figure shows projectedpositions of target centroids for a target with sets of concentriccircles arranged about the vertices of an equilateral triangle of sidelength 0.5 inches, using a camera focal length 25 mm, pixel pitch 7.4microns and view angle 45 degrees (camera to right and above paper),aligned with the camera centerline. The 45-degree view angle results inthe foreshortening of the equilateral triangle from left to right acrossthe focal plane.

Yaw

Rotation about the z-axis is designated as yaw; a positive or “righthanded” rotation about this axis (head leaning to subject's rightshoulder) results in a counterclockwise rotation of the target. Becausethis rotation usually occurs about a point lower in the neck, it istypically accompanied by a translation to the subject's right side(camera left), as seen in FIG. 12. Projected positions of targetcentroids for same conditions as FIG. 11, but before and after inducinga yaw of 15 degrees.

The median of the centered target triangle (as shown at the right inFIG. 12) is aligned approximately with a single column of camera pixels,but is rotated out of this alignment (as shown at the left side of FIG.12) by yaw. For the camera, lens, target distance and target sizedescribed above, a yaw of only 0.1 degrees results in a relativedisplacement of 0.13 pixel columns between the two ends of the median.Assuming that the centroid algorithm is able to determine position ofthe triangle vertices to 0.1 pixels in row and column, the yaw angledetermination is measurable down to and accurate to about 0.1 degrees.

Pitch

Rotation about the y-axis is designated as pitch; a positive or“right-handed” rotation about this axis (head tipped back) results inmotion of the target upward off the gantry (+z) and toward the top ofthe head (+x). For a single camera this projection is not easilydistinguishable from a simultaneous target displacement in x and y (seeFIG. 7), but for two cameras at opposite sides of the head the apparenty-displacement is in the opposite direction, removing this degeneracy. Asecond degeneracy with pitch rotation remains, for simultaneous targettranslation in +x and +z—this is discussed in more detail later—but thetilt of the target plane during a pitch rotation yields a smalldifference in the amount of translation of the base of the targettriangle relative to its apex, thus resulting in a slight apparentrotation of the target triangle as shown in FIG. 7, which is not acharacteristic of simple translation. This becomes in some embodimentsthe defining characteristic of pitch motion.

FIGS. 13A and 13B show the projected positions of target centroids forsame conditions as FIG. 11, but before and after a target pitch of 8degrees. Left is view from a camera at the left side and above thepaper, right is view from a camera at the right side and above thepaper. In each case motion away from the gantry (+z) makes the targetappear more distant from the observer.

Roll

Rotation about the x-axis is designated as roll; a positive or“right-handed” rotation about this axis (head pointing toward subject'sleft side) results in a motion of the target toward the subject's left(−y). For a single camera this motion is not easily distinguishable froma displacement in y (see FIG. 8), but for two cameras the difference inposition and in apparent foreshortening of the triangle is much morepronounced for rotation than for translation. This is because the rollmoves the target plane closer to normal incidence with one camerasightline and further from normal incidence with the other camerasightline, at a rate which is much larger than that for a simpletranslation (or yaw). There is a significant degeneracy between roll andsimultaneous +y and +z translation which is only resolved comparing thelengths of the triangle base as seen between the two cameras. A largedifference in the base lengths is a characteristic of roll motions andnot a characteristic of y+z translation, hence this is thedistinguishing characteristic for roll.

As shown in FIGS. 14A and 14B the projected positions of targetcentroids for same conditions as for FIG. 11, but before and aftertarget roll of 12 degrees. Left is view from a camera at the left sideand above the paper, right is view from a camera at the right side andabove the paper. The camera at the left side sees much wider trianglebecause target plane is closer to normal to this camera sightline. Thecamera at the left also sees much larger displacement of trianglecenter.

X-Axis Translation

Translation along the x-axis (positive toward top of head) results in amotion of the target along the vertical direction of the camera focalplane (see FIG. 15). Unlike for pitch rotation (which also involves atranslation in z), the target does not move significantly between pixelcolumns, and rotation of the target triangle is minimal. This up-downcamera translation without accompanying rotation is the distinguishingcharacteristic of x-axis translation. FIG. 15 shows the projectedpositions of target centroids for same conditions as for FIG. 11, butbefore and after target translation of 12 mm in x.

Y-Axis Translation

Translation along the y-axis (positive toward subject's right side)results in a motion of the target along the horizontal axis of thecamera focal plane (see FIGS. 16A and 16B). Unlike for roll (which alsoinvolves a differential rotation of the target plane as seen by the leftand right side cameras), the target's projected size, displacement androtation varies only slightly between left and right camera views fory-axis translation; this is the distinguishing characteristic fory-displacement. FIGS. 16A and 16B show projected positions of targetcentroids for same conditions as FIG. 11, but before and after targettranslation of 15 mm along y-axis. Left is view from a camera at theleft side and above the paper, right is view from a camera at the rightside and above the paper. Unlike roll, target displacements and sizesare similar for two cameras viewing y-axis translation.

Z-Axis Translation

Translation along the z-axis (positive toward the ceiling) results inapparent motion of the target along the horizontal axis of the camerafocal plane. Unlike for y translation, however, the direction of thehorizontal displacement is opposite between the left-side and right-sidecameras (see FIGS. 17A and 17B). This is the distinguishingcharacteristic for z-axis translation. FIGS. 17A and 17B show projectedpositions of target centroids for same conditions as for FIG. 11, butbefore and after target translation of 15 mm along the z-axis. Left isview from a camera at the left side and above the paper, right is viewfrom a camera at the right side and above the paper. Unlike translationalong y, apparent target displacement is in opposite direction in twocamera views.

Non-Degenerate Target Motion Parameters Pitch Versus (X+Z) TranslationDegeneracy

Pitch is nearly degenerate with simultaneous x and z translation, exceptfor a small tilt in the triangle vertical which results from the tilt ofthe target plane about the y axis. This tilt creates an apparentclockwise rotation of the triangle from the left-side view and anapparent counterclockwise rotation from the right side view, as shown inFIGS. 18A and 18B. These drawings show projected positions of targetcentroids for same conditions as FIG. 11, but before and after targetpitch of 4 degrees and translations in x and z of −9.5 mm and +7.8 mmrespectively. FIG. 18A is view from a camera at the left side and abovethe paper, FIG. 18B is view from a camera at the right side and abovethe paper. The camera at left sees triangle rotated clockwise, withupper vertices rotated away from the camera because of an increase in zrelative to the lower vertex. The camera at the right sees trianglerotated counterclockwise for the same reason. For a pitch motion of 0.1degrees accompanied by translations in x and z of −0.244 mm and 0.187 mmrespectively, the triangle apex centroid does not move in either cameraview. However, in this case, the left-side camera sees the triangle basedisplaced by 0.13 pixels to the right while the right-side camera seesthe triangle base displaced by 0.13 pixels to the left. Assuming thecentroiding routine can locate the vertices of the target triangle to anaccuracy of 0.1 pixels, a pitch as small as 0.1 degrees isdistinguishable from a simple translation by comparison of the verticaltilts.

Roll Versus (Y+Z) Translation Degeneracy

Roll is nearly degenerate with simultaneous y and z translation, exceptfor larger camera-to-camera differences in apparent target sizeencountered with roll, resulting from tilt of the target's plane aboutthe x-axis. A significant difference in the apparent length of thetarget triangle base is a reliable distinguishing characteristic of rollmotion rather than simple translation. FIGS. 19A and 19B show projectedpositions of target centroids for the same conditions as in FIG. 11, butbefore and after target roll of 4 degrees and translations in y and z of9.75 mm and 0.34 mm respectively. FIG. 19A is view from a camera at theleft side and above the paper, FIG. 19B is view from a camera at theright side and above the paper. Camera at left sees triangle base shrinkdue to rotation about the x-axis away from camera normal, while cameraat right sees triangle base grow due to rotation toward camera normal.

For a roll of 0.1 degrees and translations in y and z of −0.244 mm and0.0002 mm respectively, the lower centroid is unchanged in both cameraviews. In this case, the left-side camera sees the target triangle baseshrink by 0.15 pixels while the right-side camera sees the triangle basegrow by 0.15 pixels. Assuming the centroiding routine can locate thetarget centroids to an accuracy of 0.1 pixels, shifts of 0.14 pixelsshould be discernible, so a pitch as small as 0.1 degrees isdistinguishable from a simple translation by comparison of the length ofthe target triangle base.

Six-Degree-of-Freedom Motion Determination Algorithm ArchitectureComplementary Projections Versus Stereo Imaging

The target size, rotation angles and translation vector determine therelative displacement of the three target centroids precisely in threedimensions. Precise knowledge of camera and lens parameters (e.g., pixelpitch, lens focal length and radial distortion, camera location andorientation relative to nominal target position), are then sufficient topredict the location of the target centroid projections to better than0.1 pixels in row and column for each camera. In principle, the inverseproblem should be equally simple; the stereo view of the centroidprojections determine the 3D location in space of each target centroid,and the 6-DOF displacement vector can then be determined through asimple matrix inversion. In practice, however, this approach leads toexpensive and complicated requirements on camera pixel density, pixelcount, camera alignment and camera calibration. An alternate unfoldingapproach dispenses with stereo ranging and uses the two cameraprojections separately without strict requirements on precise matchingof camera alignment and magnification, to determine the 6-DOFdisplacement vector to within 0.1 degrees in each rotation angle and 0.1mm along each translation axis. This approach relaxes the pixel densityand count requirements by about a factor of 20 relative to the stereoapproach, resulting in significant savings in cost and processing speed.

Even for this 2D approach some basic measurements can be made tocalibrate camera parameters once the cameras are integrated with thehead coil; these can be easily performed at the manufacturing facility.These measurements include the projected pixel location of a singlereference point on both cameras, as well as the camera magnificationfactors for pixel displacement per degree of rotation in pitch, yaw androll, and per mm of translation along x, y and z. However, as statedbefore, it is not necessary that the cameras be exactly aligned in space(e.g. perfectly normal) or that their magnifications (lens focal lengthand distance to reference point) be identical, as has been easilyverified by simulation.

Inversion Equations

The 2D inversion problem is somewhat more difficult than the 3D problem,in that after the target centroid projections in focal plane row andcolumn are determined, significant degeneracies remain in the unfoldingmatrices for each camera. Combining the data from both cameras removesthese degeneracies through a series of interrelated, nonlinearequations. The fastest procedure for solving this inverse problem isobtained by a variant of the Newton-Raphson method, whereby anapproximate first-order solution is proposed and tested against theknown (measured) centroid locations on the two camera focal planes. Theresidual error is divided by the local derivatives with respect to eachcomponent of rotation and translation, to determine an iterativecorrection. The first-order solution is chosen by considering thefeatures of the projected target pattern which are most stronglyaffected by a single rotation angle or displacement, and linearizing theinversion problem along these feature axes.

6-DOF Extraction Algorithm

Below is described one embodiment of a method for extracting the 6degree of freedom displacement matrix from the observed target locationon two focal plane cameras. Other embodiments may be used consistentwith the disclosure herein. Further, although the embodiment belowrefers to steps, in some embodiments not all steps are included,additional steps are included, and/or the steps are not always performedin a set order.

Step 1: Characterizing the Target Images

The optical target consists of elliptical targets shown in FIG. 2A thatare drawn so as to appear as circular patterns when imaged at 45 degreesby the two cameras shown in FIG. 8A. The center of each of the threecircular patterns define one of the three vertices of an equilateraltriangle at the focal plane of each of the two cameras. A centroidcalculation routine determines the positions of the centroids at each ofthe three vertices, on each of two independent cameras. These centroidsare displayed on a computer monitor displaying the 640×480 pixels ofeach of the two cameras. FIG. 11 shows the three vertices beingdisplayed on one of the cameras. These vertex positions are (X_(i,j),Y_(i,j)) designated for vertex index i from 1 to 3, and camera index jfrom 1 to 2, resulting in twelve measured coordinates. From the twelvemeasured coordinates, and initialized values of these coordinates, sixprincipal quantities are computed to characterize the two camera viewsof the equilateral triangle target:

-   -   a) Σ_(HD)—the sum of the horizontal displacements (in pixels) of        the target center on camera 1 and camera 2; the formula used is

${\sum\limits_{i = 1}^{3}\; {\sum\limits_{j = 1}^{2}\; \left( {X_{i,j} - X_{{0\; i},j}} \right)}},$

where is X_(0i,j) the initial (zero displacement) horizontal cameracoordinate of each centroid projection.

b) Δ_(HD)—the difference between the horizontal displacements (inpixels) of the target center for camera 1 and camera 2; the formula usedis

${\sum\limits_{i = 1}^{3}\; \left( {X_{i,1} - X_{{0\; i},1}} \right)} - {\left( {X_{i,2} - X_{{0\; i},2}} \right).}$

c) Σ_(VD)—the sum of the vertical displacements (in pixels) of thetarget center for camera 1 and camera 2; the formula used is

${\sum\limits_{i = 1}^{3}\; {\sum\limits_{j = 1}^{2}\; \left( {Y_{i,j} - Y_{{0\; i},j}} \right)}},$

where Y_(0i,j) is the initial (zero displacement) vertical cameracoordinate of each centroid projection.

d) Δ_(BL)—the difference in the apparent base length of the targettriangle (in pixels) for camera 1 and camera 2; the formula used is

$\left\{ {\sqrt{\left( {X_{3,1} - X_{1,1}} \right)^{2} + \left( {Y_{3,1} - Y_{1,1}} \right)^{2}} - \sqrt{\left( {X_{3,2} - X_{1,1}} \right)^{2} + \left( {Y_{3,2} - Y_{1,2}} \right)^{2}}} \right\}.$

e) Σ_(MT)—the sum of the apparent median tilt of the target triangle(offset in horizontal pixels between center-of-base and apex) for camera1 and camera 2; the formula used is

$\sum\limits_{j = 1}^{2}\; {\left\{ {\left( {X_{2,j} - \frac{X_{3,j} + X_{1,j}}{2}} \right) - \left( {X_{02,j} - \frac{X_{03,j} + X_{01,j}}{2}} \right)} \right\}.}$

f) Δ_(MT)—the difference between the apparent median tilt of the targettriangle (in pixels) for camera 1 and camera 2; the formula used is

$\left\{ {\left( {X_{2,1} - \frac{X_{3,1} + X_{1,1}}{2}} \right) - \left( {X_{02,1} - \frac{X_{03,1} + X_{01,1}}{2}} \right)} \right\} - \left\{ {\left( {X_{2,2} - \frac{X_{3,2} + X_{1,2}}{2}} \right) - \left( {X_{02,2} - \frac{X_{03,2} + X_{01,2}}{2}} \right)} \right\}$

Step 2: Characterizing Global Variation in Principal Quantities with6-DOF Motions

Partial derivatives relative to subject displacements and rotations (φ,θ, ψ, Δx, Δy, Δz), of the principal quantities described above, aboutthe initial (non-displaced) position, are computed numerically. Here:

Roll φ is right-handed rotation about the x-axis

Pitch θ is right-handed rotation about the y-axis

Yaw ψ is right-handed rotation about the z-axis

Δx is toe-to-head direction

Δy is left-to-right direction

Δz is down-to-up direction

Starting from an initial target position in 3-D world space, defined as(φ, θ, ψ, Δx, Δy, Δz)=(0, 0, 0, 0, 0, 0), the initial target vertexworld coordinates (x_(0i),y_(0i),z_(0i)) are determined for vertex indexi=1 to 3, based on the geometric size and shape of the target triangleand definition of a convenient coordinate origin.

Local partial derivatives of each of the principal quantities, withrespect to each of the 6 degrees of freedom (roll, pitch, yaw, dx, dy,dz), are performed numerically by evaluating changes in these quantitiesfor small increments in each degree of freedom. Changes in the targetvertex positions for specified motions along the six degrees of freedomare computed using the Euler rotation matrices and translation vector:

$\begin{matrix}{{{\begin{pmatrix}x_{i} \\y_{i} \\z_{i}\end{pmatrix} = {{\begin{pmatrix}1 & 0 & 0 \\0 & {\cos \; \phi} & {{- \sin}\; \phi} \\0 & {\sin \; \phi} & {\cos \; \phi}\end{pmatrix}\begin{pmatrix}{\cos \; \theta} & 0 & {\sin \; \theta} \\0 & 1 & 0 \\{{- \sin}\; \theta} & 0 & {\cos \; \theta}\end{pmatrix}\begin{pmatrix}{\cos \; \psi} & {{- \sin}\; \psi} & 0 \\{\sin \; \psi} & {\cos \; \psi} & 0 \\0 & 0 & 1\end{pmatrix}\begin{pmatrix}x_{oi} \\y_{oi} \\z_{oi}\end{pmatrix}} +}}\quad}\begin{pmatrix}{\Delta \; x} \\{\Delta \; y} \\{\Delta \; z}\end{pmatrix}} & \lbrack 1\rbrack\end{matrix}$

Subsequently, the camera projections of these new target vertexpositions are determined using a geometric projection calculation. Givenaccurate knowledge of camera positions, pixel size and lens focallength, the horizontal and vertical pixel numbers on each camera focalplane (camera index j equal to 1 or 2) that these new 3-D positions inspace should project onto is as follows:

$\begin{matrix}{{X_{i,j} = {X_{0,j} + {\left( \frac{f.l.}{s_{pix}} \right)\sin \; \alpha_{i,j}\cos \; \beta_{i,j}}}},{Y_{i,j} = {Y_{0,j} + {\left( \frac{f.l.}{s_{pix}} \right)\sin \; \alpha_{i,j}\sin \; \beta_{i,j}}}}} & \lbrack 2\rbrack\end{matrix}$

Here and X_(i,j) are Y_(i,j) the horizontal and vertical pixel numbersfor translated target vertex i projected onto the camera j sensor,X_(0,j) and Y_(0,j) are the horizontal and vertical number of the pixelcolumn and row intersected by the optical axis of that camera (typicallyat or very near the camera center), f.l. and s_(pix) are the lens focallength and camera pixel pitch, and the angles α_(i,j) and β_(i,j) arethe polar and azimuth angles locating target vertex i, relative to thecamera j focal axis. These angles are calculated from the vertex worldcoordinates as follows:

$\begin{matrix}{{\alpha_{i,j} = {\sin^{- 1}\left( \frac{\sqrt{\left( {x_{\bot{,j}} - x_{i}} \right)^{2} + \left( {y_{\bot{,j}} - y_{i}} \right)^{2} + \left( {z_{\bot{,j}} - z_{i}} \right)^{2}}}{\sqrt{\left( {x_{cj} - x_{i}} \right)^{2} + \left( {y_{cj} - y_{i}} \right)^{2} + \left( {z_{cj} - z_{i}} \right)^{2}}} \right)}},} & \lbrack 3\rbrack \\{{\beta_{i,j} = {\pm {\cos^{- 1}\left( \frac{{\left( {x_{\bot{,j}} - x_{i}} \right)\left( {y_{cj} - y_{0}} \right)} - {\left( {y_{\bot{,j}} - y_{i}} \right)\left( {x_{cj} - x_{0}} \right)}}{\begin{matrix}\sqrt{\left( {x_{\bot{,j}} - x_{i}} \right)^{2} + \left( {y_{\bot{,j}} - y_{i}} \right)^{2} + \left( {z_{\bot{,j}} - z_{i}} \right)^{2}} \\\sqrt{\left( {x_{cj} - x_{0}} \right)^{2} + \left( {y_{cj} - y_{0}} \right)^{2}}\end{matrix}} \right)}}},} & \lbrack 4\rbrack\end{matrix}$

where the point (x_(⊥,j),y_(⊥,j),z_(⊥,j)) is the point of intersectionbetween the camera optical axis and the plane perpendicular to theoptical axis which includes the translated target vertex(x_(⊥)i,y_(⊥)i,z_(⊥)i):

x _(⊥,j) =x ₀+κ(x _(cj) −x ₀);y _(⊥,j) =y ₀+κ(y _(cj) −y ₀);z _(⊥,j) =z₀+κ(z _(cj) −z ₀),  [5]

with (x_(⊥)cj,y_(⊥)cj,z_(⊥)cj) defining the 3-D position of camera j,(x_(⊥)0,y_(⊥)0,z_(⊥)0) defining the nominal boresight position of bothcameras at the un-displaced target center and the constant κ based ongeometric projection and given by:

$\begin{matrix}{\kappa = {\left\{ \frac{{\left( {x_{cj} - x_{0}} \right)\left( {x_{i} - x_{0}} \right)} + {\left( {y_{cj} - y_{0}} \right)\left( {y_{i} - y_{0}} \right)} + {\left( {z_{cj} - z_{0}} \right)\left( {z_{i} - z_{0}} \right)}}{\left( {x_{cj} - x_{0}} \right)^{2} + \left( {y_{cj} - y_{0}} \right)^{2} + \left( {z_{cj} - z_{0}} \right)^{2}} \right\}.}} & \lbrack 6\rbrack\end{matrix}$

In equation [4], the inverse cosine function is taken to range from 0 toπ, and the appropriate sign for β_(i,j) is given by:

sign[β_(i,j)]=sign[(z _(cj) −z _(i)){(x _(cj) −x ₀)(x _(⊥,j) −x _(i))+(y_(cj) −y ₀)(y _(⊥,j) −y _(i))}−(z _(⊥,j) −z _(i)){(x _(cj) −x ₀)²+(y_(cj) −y ₀)²}]

During this determination of the camera projection of the 3-D targetvertices, a compensation function may be applied for large values of thepolar angle α_(i,j) to account for barrel distortion in the lens, basedon prior lens calibration measurements. The geometric value for α_(i,j)is first computed based on equation [3] and then adjusted for lensdistortion by way of a pre-determined look-up table or measured fitfunction, and this new compensated value for α_(i,j) is then used in thecalculation of X_(i,j) and Y_(i,j) through equation [2].

To numerically evaluate the partial derivatives of the principalquantities about the initialized target position, the un-displaced 3-Dtarget vertex coordinates) (x_(0i),y_(0i),z_(0i)) are first projected tocamera coordinates using equations [2] through [6] above, and initialvalues are computed for each of the principal quantities described inStep 1 (most should be zero or near-zero at the starting position). Thensmall increments of roll, pitch, yaw, x-, y- and z-axis displacementsare introduced one at a time; for each increment the new worldcoordinates and the new camera projections of the target vertices arecomputed and the principal quantities are re-calculated. The change ineach principal quantity is divided by the small angular or displacementincrement to determine the partial derivative.

For instance, to determine the partial derivatives with respect to roll,the displacement vector (φ, θ, ψ, Δx, Δy, Δz)=(ôφ, 0, 0, 0, 0, 0) isintroduced to the general displacement equation [1] to determine thetranslated target vertex positions (x_(i),y_(i),z_(i)). The conversionto camera coordinates is then performed using equations [2] through [6],and the principal quantities are calculated as outlined in Step 1. Thedifference between each principal quantity and the corresponding valueof that quantity for the un-displaced calculation is divided by thesmall increment in roll, to give the partial derivative of each quantitywith respect to roll. To determine partial derivatives with respect topitch, the displacement vector (φ, θ, ψ, Δx, Δy, Δz)=(0, δθ, 0, 0, 0, 0)is used to initiate the calculations, and so on for all six degrees offreedom.

Each of these six repetitions produces one column of the global partialderivative matrix:

${\begin{pmatrix}\frac{\partial\Sigma_{HD}}{\partial\phi} & \frac{\partial\Sigma_{HD}}{\partial\theta} & \frac{\partial\Sigma_{HD}}{\partial\psi} & \frac{\partial\Sigma_{HD}}{\partial x} & \frac{\partial\Sigma_{HD}}{\partial y} & \frac{\partial\Sigma_{HD}}{\partial z} \\\frac{\partial\Delta_{HD}}{\partial\phi} & \frac{\partial\Delta_{HD}}{\partial\theta} & \frac{\partial\Delta_{HD}}{\partial\psi} & \frac{\partial\Delta_{HD}}{\partial x} & \frac{\partial\Delta_{HD}}{\partial y} & \frac{\partial\Delta_{HD}}{\partial z} \\\frac{\partial\Sigma_{VD}}{\partial\phi} & \frac{\partial\Sigma_{VD}}{\partial\theta} & \frac{\partial\Sigma_{VD}}{\partial\psi} & \frac{\partial\Sigma_{VD}}{\partial x} & \frac{\partial\Sigma_{VD}}{\partial y} & \frac{\partial\Sigma_{VD}}{\partial z} \\\frac{\partial\Delta_{BL}}{\partial\phi} & \frac{\partial\Delta_{BL}}{\partial\theta} & \frac{\partial\Delta_{BL}}{\partial\psi} & \frac{\partial\Delta_{BL}}{\partial x} & \frac{\partial\Delta_{BL}}{\partial y} & \frac{\partial\Delta_{BL}}{\partial z} \\\frac{\partial\Sigma_{MT}}{\partial\phi} & \frac{\partial\Sigma_{MT}}{\partial\theta} & \frac{\partial\Sigma_{MT}}{\partial\psi} & \frac{\partial\Sigma_{MT}}{\partial x} & \frac{\partial\Sigma_{MT}}{\partial y} & \frac{\partial\Sigma_{MT}}{\partial z} \\\frac{\partial\Delta_{MT}}{\partial\phi} & \frac{\partial\Delta_{MT}}{\partial\theta} & \frac{\partial\Delta_{MT}}{\partial\psi} & \frac{\partial\Delta_{MT}}{\partial x} & \frac{\partial\Delta_{MT}}{\partial y} & \frac{\partial\Delta_{MT}}{\partial z}\end{pmatrix}}_{({0,0,0,0,0,0})}.$

Step 3: Determining First-Order Displacement Vector

A first-order approximation to the displacement matrix is determined bymultiplying the matrix of measured principal quantities, as determinedin Step 1, by the inverse of the partial derivative matrix computed inStep 2:

$\begin{pmatrix}\phi_{0} \\\theta_{0} \\\psi_{0} \\\left( {\Delta \; x} \right)_{0} \\\left( {\Delta \; y} \right)_{0} \\\left( {\Delta \; z} \right)_{0}\end{pmatrix} = {\begin{pmatrix}\frac{\partial\Sigma_{HD}}{\partial\phi} & \frac{\partial\Sigma_{HD}}{\partial\theta} & \frac{\partial\Sigma_{HD}}{\partial\psi} & \frac{\partial\Sigma_{HD}}{\partial x} & \frac{\partial\Sigma_{HD}}{\partial y} & \frac{\partial\Sigma_{HD}}{\partial y} \\\frac{\partial\Delta_{HD}}{\partial\phi} & \frac{\partial\Delta_{HD}}{\partial\theta} & \frac{\partial\Delta_{HD}}{\partial\psi} & \frac{\partial\Delta_{HD}}{\partial x} & \frac{\partial\Delta_{HD}}{\partial y} & \frac{\partial\Delta_{HD}}{\partial z} \\\frac{\partial\Sigma_{VD}}{\partial\phi} & \frac{\partial\Sigma_{VD}}{\partial\theta} & \frac{\partial\Sigma_{VD}}{\partial\psi} & \frac{\partial\Sigma_{VD}}{\partial x} & \frac{\partial\Sigma_{VD}}{\partial y} & \frac{\partial\Sigma_{VD}}{\partial z} \\\frac{\partial\Delta_{BL}}{\partial\phi} & \frac{\partial\Delta_{BL}}{\partial\theta} & \frac{\partial\Delta_{BL}}{\partial\psi} & \frac{\partial\Delta_{BL}}{\partial x} & \frac{\partial\Delta_{BL}}{\partial y} & \frac{\partial\Delta_{BL}}{\partial z} \\\frac{\partial\Sigma_{MT}}{\partial\phi} & \frac{\partial\Sigma_{MT}}{\partial\theta} & \frac{\partial\Sigma_{MT}}{\partial\psi} & \frac{\partial\Sigma_{MT}}{\partial x} & \frac{\partial\Sigma_{MT}}{\partial y} & \frac{\partial\Sigma_{MT}}{\partial z} \\\frac{\partial\Delta_{MT}}{\partial\phi} & \frac{\partial\Delta_{MT}}{\partial\theta} & \frac{\partial\Delta_{MT}}{\partial\psi} & \frac{\partial\Delta_{MT}}{\partial x} & \frac{\partial\Delta_{MT}}{\partial y} & \frac{\partial\Delta_{MT}}{\partial z}\end{pmatrix}^{- 1}{\begin{pmatrix}\Sigma_{HD} \\\Delta_{HD} \\\Sigma_{VD} \\\Delta_{BL} \\\Sigma_{MT} \\\Delta_{MT}\end{pmatrix}.}}$

Step 4: Characterizing Local Variation in Principal Quantities with6-DOF Motions

First order values for (φ, θ, ψ, Δx, Δy, Δz) determined in Step 3 areentered into the translation equation [1] to determine the correspondingtranslated 3-D target position (x_(⊥)i,y_(⊥)i,z_(⊥)i) for each of thethree target vertices. These world coordinates are projected to cameracoordinates (X_(i,j), Y_(i,j)) using equations [2] through [6], and theprincipal quantities are re-calculated. These six quantities arecompared against the measured values of these quantities determined inStep 1, to create a residual error matrix:

(σ_(Σ) _(HD) ,σ_(Δ) _(HD) ,σ_(Σ) _(VD) ,σ_(Δ) _(BL) ,σ_(Σ) _(MT) ,σ_(Δ)_(MT) ).

Local partial derivatives of the principal quantities are calculated byintroducing small increments in roll, pitch, yaw, x-, y- and z-axisdisplacements one at a time as before, but this time the increments arerelative to the first-order displacement vector. For each increment, thenew world coordinates and the new camera projections of the targetvertices are re-computed and the principal quantities are re-calculated.The change in each principal quantity is divided by the small angular ordisplacement increment to determine a local partial derivative. Forinstance, to calculate partial derivatives with respect to roll, thefirst-order displacement vector {φ₀, θ₀, ψ₀, (Δx)₀, (Δy)₀, (Δz)₀} isreplaced by {φ₀+ôφ, θ₀, ψ₀, (Δx)₀, (Δy)₀, (Δz)₀} and resulting changesto each of the principal quantities is divided by ôφ to determine thelocal derivative with respect to roll. This is repeated for each of thesix degrees of freedom.

Each of these six repetitions produces one column of the new localpartial derivative matrix:

${\begin{pmatrix}\frac{\partial\Sigma_{HD}}{\partial\phi} & \frac{\partial\Sigma_{HD}}{\partial\theta} & \frac{\partial\Sigma_{HD}}{\partial\psi} & \frac{\partial\Sigma_{HD}}{\partial x} & \frac{\partial\Sigma_{HD}}{\partial y} & \frac{\partial\Sigma_{HD}}{\partial z} \\\frac{\partial\Delta_{HD}}{\partial\phi} & \frac{\partial\Delta_{HD}}{\partial\theta} & \frac{\partial\Delta_{HD}}{\partial\psi} & \frac{\partial\Delta_{HD}}{\partial x} & \frac{\partial\Delta_{HD}}{\partial y} & \frac{\partial\Delta_{HD}}{\partial z} \\\frac{\partial\Sigma_{VD}}{\partial\phi} & \frac{\partial\Sigma_{VD}}{\partial\theta} & \frac{\partial\Sigma_{VD}}{\partial\psi} & \frac{\partial\Sigma_{VD}}{\partial x} & \frac{\partial\Sigma_{VD}}{\partial y} & \frac{\partial\Sigma_{VD}}{\partial z} \\\frac{\partial\Delta_{BL}}{\partial\phi} & \frac{\partial\Delta_{BL}}{\partial\theta} & \frac{\partial\Delta_{BL}}{\partial\psi} & \frac{\partial\Delta_{BL}}{\partial x} & \frac{\partial\Delta_{BL}}{\partial y} & \frac{\partial\Delta_{BL}}{\partial z} \\\frac{\partial\Sigma_{MT}}{\partial\phi} & \frac{\partial\Sigma_{MT}}{\partial\theta} & \frac{\partial\Sigma_{MT}}{\partial\psi} & \frac{\partial\Sigma_{MT}}{\partial x} & \frac{\partial\Sigma_{MT}}{\partial y} & \frac{\partial\Sigma_{MT}}{\partial z} \\\frac{\partial\Delta_{MT}}{\partial\phi} & \frac{\partial\Delta_{MT}}{\partial\theta} & \frac{\partial\Delta_{MT}}{\partial\psi} & \frac{\partial\Delta_{MT}}{\partial x} & \frac{\partial\Delta_{MT}}{\partial y} & \frac{\partial\Delta_{MT}}{\partial z}\end{pmatrix}}_{\{\begin{matrix}{\phi_{0},\theta_{0},\psi_{0},{({\Delta \; x})}_{0},} \\{{({\Delta \; y})}_{0},{({\Delta \; z})}_{0}}\end{matrix}\}} {\quad.}$

Step 5: Determining Coarse Correction to First-Order Displacement Vector

A coarse correction is computed to improve the first-order displacementvector and reduce residual error, by multiplying the residual errormatrix determined in Step 4 by the inverse of the local partialderivative matrix, also determined in Step 4:

$\begin{pmatrix}{\Delta \; \phi} \\{\Delta \; \theta} \\{\Delta \; \psi} \\{\Delta \left( {\Delta \; x} \right)} \\{\Delta \left( {\Delta \; y} \right)} \\{\Delta \left( {\Delta \; z} \right)}\end{pmatrix} = {\begin{pmatrix}\frac{\partial\Sigma_{HD}}{\partial\phi} & \frac{\partial\Sigma_{HD}}{\partial\theta} & \frac{\partial\Sigma_{HD}}{\partial\psi} & \frac{\partial\Sigma_{HD}}{\partial x} & \frac{\partial\Sigma_{HD}}{\partial y} & \frac{\partial\Sigma_{HD}}{\partial y} \\\frac{\partial\Delta_{HD}}{\partial\phi} & \frac{\partial\Delta_{HD}}{\partial\theta} & \frac{\partial\Delta_{HD}}{\partial\psi} & \frac{\partial\Delta_{HD}}{\partial x} & \frac{\partial\Delta_{HD}}{\partial y} & \frac{\partial\Delta_{HD}}{\partial z} \\\frac{\partial\Sigma_{VD}}{\partial\phi} & \frac{\partial\Sigma_{VD}}{\partial\theta} & \frac{\partial\Sigma_{VD}}{\partial\psi} & \frac{\partial\Sigma_{VD}}{\partial x} & \frac{\partial\Sigma_{VD}}{\partial y} & \frac{\partial\Sigma_{VD}}{\partial z} \\\frac{\partial\Delta_{BL}}{\partial\phi} & \frac{\partial\Delta_{BL}}{\partial\theta} & \frac{\partial\Delta_{BL}}{\partial\psi} & \frac{\partial\Delta_{BL}}{\partial x} & \frac{\partial\Delta_{BL}}{\partial y} & \frac{\partial\Delta_{BL}}{\partial z} \\\frac{\partial\Sigma_{MT}}{\partial\phi} & \frac{\partial\Sigma_{MT}}{\partial\theta} & \frac{\partial\Sigma_{MT}}{\partial\psi} & \frac{\partial\Sigma_{MT}}{\partial x} & \frac{\partial\Sigma_{MT}}{\partial y} & \frac{\partial\Sigma_{MT}}{\partial z} \\\frac{\partial\Delta_{MT}}{\partial\phi} & \frac{\partial\Delta_{MT}}{\partial\theta} & \frac{\partial\Delta_{MT}}{\partial\psi} & \frac{\partial\Delta_{MT}}{\partial x} & \frac{\partial\Delta_{MT}}{\partial y} & \frac{\partial\Delta_{MT}}{\partial z}\end{pmatrix}^{- 1}{\begin{pmatrix}\sigma_{\Sigma_{HD}} \\\sigma_{\Delta_{HD}} \\\sigma_{\Sigma_{VD}} \\\sigma_{\Delta_{BL}} \\\sigma_{\Sigma_{MT}} \\\sigma_{\Delta_{MT}}\end{pmatrix}.}}$

The first-order displacement vector is incremented by the coarsecorrection matrix to create a better approximation to the displacementvector:

{φ₀+Δφ,θ₀+Δθ,ψ₀+Δψ,(Δx)₀+Δ(Δx),(Δy)₀+Δ(Δy),(Δz)₀+Δ(Δz)}

Step 6: Performing Fine Correction to Determine Final 6DOF DisplacementVector

Steps 4 and 5 are repeated, starting with the coarse-correcteddisplacement vector, to determine a final fine correction to thedisplacement vector. After this iteration, the resultant fine correctionincrements are added to the coarse-corrected vector to create the final6-DOF displacement vector. Empirical results from a general simulationindicate that this fine correction is sufficient in all cases to reduceresidual errors to well below the stated 0.1-degree, 0.1-mm tolerances.

Algorithm Numerical Simulation to Verify Absolute Convergence

As cameras, targets and rotation stages are being procured andassembled, the 6 DOF decomposition algorithm can be coded and tested fora test set of rotations. It is clear that the routine will converge forsmall translations and rotations, but it can be potentially advantageousto determine whether there are limitations on its convergence forextreme displacements in all six degrees of freedom. To this end, weimagine an extreme target displacement, calculate the 3D position of thedisplaced target, calculate the centroid positions that will be seen oneach of the two cameras, and run the decomposition algorithm todetermine speed of convergence.

In some embodiments, to demonstrate absolute convergence of theiterative 6DOF unfolding algorithm, the simulation is started with atest set of very large rotations and displacements, as listed in Table 1below.

TABLE 1 Example of a set of extreme angular rotations and lineartranslations of an imaginary patient for purposes of testing algorithmconvergence. Head Yaw (Lean) Psi (deg toward patient's right shoulder)8.0000 Head Pitch (Nod) Theta (deg relative to level; pos is toward15.0000 top of head) Head Roll (Shake) Phi (deg relative to square; postoward 12.0000 patient's left side) Head shift dx (mm toward top ofhead) −9.0000 Head shift dy (mm to patient's right) 3.0000 Head shift dz(mm away from table) 7.0000

The simulation begins by determining the locations of the displacedcentroids that will be seen by each camera, allowing for some degree ofmispointing and misalignment of each camera. The original (nominal)target location is rotated and displaced by the Euler rotation formalismpresented in Section 2.5.2.2, to determine the three displaced targetcentroid locations in three-dimensional space. Next these “worldcoordinates” are translated to 2-D “camera coordinates” for each of thetwo cameras independently, as described in the same Section.

Assuming the target is imaged into these camera coordinates, but thatthe operator has no prior knowledge of the displacement matrix givingrise to this target position, we use the algorithm as described inSection 2.5.2 from end to end to recreate the displacement matrix. Bythe end of Step 3 (Section 2.5.2.3), the algorithm returns an initialestimate of the 6DOF displacement vector, as shown in Table 2 below.

TABLE 2 First estimate of 6DOF displacement based on method described inSection 2.5.2. First Approximation Yaw (degrees) 4.4313 FirstApproximation Pitch (degrees) −19.4474 First Approximation Roll(degrees) 8.8784 First Approximation X displacement −6.4257 (mm) FirstApproximation Y displacement −2.5639 (mm) First Approximation Zdisplacement −5.9428 (mm)

As expected, residual errors at this stage are atypically large, due tothe extreme magnitudes of the translations and rotations chosen for thissimulation along and about each axis; this situation creates a good testfor absolute convergence of the Newton Raphson algorithm methodology.Assuming this estimate to be correct, the algorithm in Step 4 (Section2.5.2.4) again calculates the displaced position of the target, theresulting centroid positions seen by each camera, and the principalquantities (vertical tip sum and difference, base length difference,vertical displacement sum, and horizontal displacement sum anddifference) which would result, for comparison with the actual observedvalues. The residual errors, in pixels, and the local derivatives ofeach of the principal values for small changes (pixels per 0.1 degrees)in yaw, pitch, and roll, and for small changes (pixels per 0.1 mm) indx, dy and dz are calculated as described in Section 2.5.2.4, andtabulated as shown in Table 3 below.

TABLE 3 Residual Error (in pixels) and local derivatives with respect toYaw, Pitch, Roll (pixels per 0.1 deg), x-displacement, y-displacement,and z-displacement (pixels per 0.1 mm), of the principal quantitiesVertical Tip Sum, Vertical Tip Difference, Base Length Difference,Vertical Displacement Sum, Horizontal Displacement Sum, and HorizontalDisplacement Difference. Residual ∂/∂Y ∂/∂P ∂/∂R ∂/∂x ∂/∂y ∂/∂z ErrorVT1 + VT2 0.2575 0.0383 −0.099 0.0021 0.0045 0.0021 −3.6538 VT1 − VT20.0657 −0.2756 −0.0131 0.0005 0.0018 0.0274 6.8709 BL1 − BL2 −0.52230.0277 0.4988 0.0109 −0.0702 0.0105 −2.9918 VDT + VD2 −0.3118 5.81340.0350 1.8813 0.0112 −0.2223 −168.5591 HD1 + HD2 −2.5875 −0.1680 3.36510.0117 −1.3090 −0.0124 58.1859 HD1 − HD2 0.5823 1.4452 0.7697 −0.01400.1114 −1.4280 20.793

The matrix of derivatives at the left of Table 3 is inverted andmultiplied by the residual error vector at the right, to yieldfirst-order corrections to the initial estimate of the displacementvector, as described in Section 2.5.2.5, and as shown at the left ofTable 4 below. These are added to the initial estimates, to produce themore refined estimate of the 6 DOF displacement vector, shown at theright of Table 4.

TABLE 4 (left) First-Order Corrections to Initial Estimates of Yaw,Pitch, Roll, dx, dy and dz, obtained by inverting the matrix ofderivatives at left of Table 3 above and multiplying this inverse matrixby the residual error vector at right of Table 3. These corrections areadded to initial 6DOF motion estimates to produce improved estimates atright above. Yaw Adjustment 3.8632 First Newton Iteration Yaw 8.2945(deg) (deg) Pitch Adjustment 4.5672 First Newton Iteration Pitch−14.8803 (deg) (deg) Roll Adjustment 3.5642 First Newton Iteration Roll12.4426 (deg) (deg) dx Adjustment −3.0846 First Newton Iteration Delta−9.5103 (mm) X (mm) dy Adjustment 6.5969 First Newton Iteration Delta4.0329 (mm) Y (mm) dz Adjustment 12.9426 First Newton Iteration Delta6.9998 (mm) Z (mm)

This process is repeated for a second and final time as described inSection 2.5.2.6, assuming again that the (now refined) 6 DOFdisplacement vector is accurate, and calculating first the 3D targetcentroid positions and then the locations of the target centroids asprojected onto each of the two camera focal planes. Again the sixprincipal quantities are computed and compared with the actualobservations to produce a vector of residual errors. Again the localderivatives are computed, this time at the location of the first-orderdisplacement vector. The results are tabulated as shown in Table 5below.

TABLE 5 First-Order Residual Error (in pixels) and new local derivativeswith respect to Yaw, Pitch, Roll (pixels per 0.1 deg), x-displacement,y-displacement, and z-displacement (pixels per 0.1 mm), of the principalquantities Vertical Tip Sum, Vertical Tip Difference, Base LengthDifference, Vertical Displacement Sum, Horizontal Displacement Sum, andHorizontal Displacement Difference. Residual ∂/∂Y ∂/∂P ∂/∂R ∂/∂x ∂/∂y∂/∂z Error VT1 + VT2 0.2498 0.0545 0.0785 0.0020 0.0028 0.0007 0.1715VT1 − VT2 0.0682 0.2935 0.0223 0.0012 0.0034 0.0212 0.0827 BL1 − BL2−0.3146 0.0536 0.4966 0.0372 0.0723 0.0094 0.5096 VD1 + VD2 −0.59275.7797 0.0405 1.9353 0.0084 0.1911 −4.3941 HD1 + HD2 −2.5162 0.32373.7395 0.0074 1.3067 0.0135 4.8578 HD1 − HD2 −0.6876 1.779 0.7547 0.0177−0.0884 −1.4784 2.5723

The matrix of derivatives at the left of Table 5 is inverted andmultiplied by the residual error vector at the right, to yield finalcorrections to the first-order estimate of the displacement vector, asshown at the left of Table 6 below. These corrections are added to thefirst-order estimates, to produce the final second-order estimate of the6 DOF displacement vector, shown at the right of Table 6.

TABLE 6 (left) Second-Order Corrections to First-Order Estimates of Yaw,Pitch, Roll, dx, dy and dz, obtained by inverting the matrix ofderivatives at left of Table 5 above and multiplying this inverse matrixby the residual error vector at right Table 5. These corrections areadded to first-order correction obtained by the same method, to producefinal values for each of the 6 DOF motions used in the simulation. YawAdjustment −0.2947 Final Yaw 7.9999 (deg) (deg) Pitch Adjustment 0.2440Final Pitch 15.0003 (deg) (deg) Roll Adjustment 0.4408 Final Roll11.9978 (deg) (deg) dx Adjustment 0.5114 Final Delta X −8.9989 (mm) (mm)dy Adjustment 1.0377 Final Delta Y 2.9952 (mm) (mm) dz Adjustment 0.0058Final Delta Z 6.994 (mm) (mm)

Even for the extreme rotations and displacements used in this model, thealgorithm is shown to converge to within 0.003 degrees and 0.006 mm inonly two iterations. Given the number of floating-point operationsneeded to perform the initial estimate and two successive iterations ofthe Newton method, the algorithm can produce a solution on a typicallaptop computer in less than 5 milliseconds.

Quaternion Representation

The head coil ICD specifies the rotation vector in terms of thequaternion, for which (still using right-handed Euler angle rotationconventions):

$q = {\begin{bmatrix}q_{r} \\q_{x} \\q_{y} \\q_{z}\end{bmatrix} = \begin{bmatrix}{{{\cos \left( {\phi/2} \right)}{\cos \left( {\theta/2} \right)}{\cos \left( {\psi/2} \right)}} - {{\sin \left( {\phi/2} \right)}{\sin \left( {\theta/2} \right)}{\sin \left( {\psi/2} \right)}}} \\{{{- {\sin \left( {\phi/2} \right)}}{\cos \left( {\theta/2} \right)}{\cos \left( {\psi/2} \right)}} - {{\cos \left( {\phi/2} \right)}{\sin \left( {\theta/2} \right)}{\sin \left( {\psi/2} \right)}}} \\{{{- {\cos \left( {\phi/2} \right)}}{\sin \left( {\theta/2} \right)}{\cos \left( {\psi/2} \right)}} + {{\sin \left( {\phi/2} \right)}{\cos \left( {\theta/2} \right)}{\sin \left( {\psi/2} \right)}}} \\{{{- {\cos \left( {\phi/2} \right)}}{\cos \left( {\theta/2} \right)}{\sin \left( {\psi/2} \right)}} - {{\sin \left( {\phi/2} \right)}{\sin \left( {\theta/2} \right)}{\cos \left( {\psi/2} \right)}}}\end{bmatrix}}$

The translation vector is unchanged from the form calculated here.

Centroid Determination Algorithm

The centroid location on the focal plane is given by:

${x_{0} = \frac{\sum\limits_{ij}{x_{ij}I_{ij}}}{\sum\limits_{ij}I_{ij}}},{y_{0} = {\frac{\sum\limits_{ij}{y_{ij}I_{ij}}}{\sum\limits_{ij}I_{ij}}.}}$

This calculation is performed for three subregions 2010 on the target asshown in FIG. 20B (the dashed lines do not appear on the real target),inverting the image such that large count numbers correspond to black(near 4095, for the 12-bit monochrome camera readout) and small countnumbers for white (near 0). With a minimal amount of sophistication, theroutine can detect the pattern of circles and approximately locate thesesubregions 2010 automatically. In some embodiments, the routine can beinitialized with a key click to identify the approximate position ofeach centroid at startup. Subsequently, the three regions of interestfor each new frame will be centered at the centroid locations from theprevious frame, plus and minus 48 pixel rows and plus and minus 48 pixelcolumns. Regions of interest around each of the three target circleswhich can be integrated to determine target centroids.

Centroid Determination

In some embodiments, a test target can be printed and mounted in theview field of a monochrome camera at an angle of approximately 45degrees. At this angle the elliptical target projected to anapproximately round target on the camera focal plane. In someembodiments the camera can be focused at a full-scale printed targetoriented at 45 degrees at a distance of 14.1 inches. Camera field ofview is roughly the size of the rectangle in the center of the cameracalibration target mounted next to the target.

The calculated target centroid is displayed as a red dot at the centerof a LabView image, and displayed as a floating point (x,y) pair to theright of the image. At illumination levels above about 20% of fullscale, the measured centroid location does not fluctuate above the0.1-pixel level in row or column; for lower intensity levels,statistical fluctuations exceed this threshold. It is noted, however,that for the black-on-white printed target, uniformity of illuminationcan be potentially important—if the target is illuminated significantlymore strongly from the left or right side, for instance, the momentcalculation could add bias in the horizontal direction and would shiftthe centroid outside of the specified error threshold. This effect couldin some cases put an undesirable cost constraint on the illuminationapproach, so an intensity thresholding algorithm is first implemented,by which the target histogram is clipped near the lower extrema for thebright and dark region intensities, eliminating the undesirable effect.In some embodiments, a Camera Control screen view can allow control ofcamera frame rate and readout resolution, showing manually-selectedregion of interest. Full camera field of view is approximatelyrepresented by a black region on the screen. The centroid can bedisplayed as a red dot at the center of the circular target, and camerax-y coordinates are displayed as floating point numbers to 2-decimalprecision to the right of the display.

Example 1 Camera Calibration

As with any camera lens, the lens used for the head tracker could havesome level of distortion as a function of distance from imaging axis.Azimuthal distortion should be negligible, but radial distortion can bemeasured after lens installation and fit to a polynomial curve to allowrapid compensation of centroid positions near the edges of the camerafield of view. The 6DOF unfolding algorithm can be constructed toaccommodate typical levels of radial distortion as a second-ordercompensation during the application of the Newton Raphson iterationmethod.

Radial distortion can be determined using a printed reference targetwith concentric circles of diameter ⅓″, ⅔″, 1″, and so on up to amaximum diameter of 4 inches, as shown in FIGS. 22A and 22B. Theapproximate FOV of the camera and 25 mm lens at a working distance of14.08 inches is 2″×2.67″, as indicated by the inner rectangle printed onthe target. The camera is mounted 14.24 inches from the target such thatthe inner rectangle is visible at the edges of the camera FOV, and thetarget is centered in this field. A single image frame is captured andthe intersections of the circles and radial lines are identified andprecisely located through local centroid calculations on the image. Thepolar angles of the world coordinate system are compared against thepolar angles recorded on the camera to determine the radial distortion.FIG. 22A is the Camera Calibration Target and FIG. 22B is the off-axisradial distortion of the 25 mm fixed-focal length camera lens, measuredby comparing the diameters of circles recorded on the camera focalplane.

In one embodiment, the measured radial distortion measured for theTechSpec High Resolution Fixed Focus 25 mm lens follows camera polarangle θ_(C)=(1+0.0053144θ−0.0016804θ²+0.0002483θ³−0.0000138θ⁴)θ, withlaboratory polar angle θ in degrees. At the extreme corner of theviewing field, where θ˜6.75°, camera aberration results in a radialgrowth in camera angle of about 0.7% relative to true angle, or about2.8 pixels in radius.

Full 6-DOF Tracking

The full 6-DOF tracking algorithm was coded in LabView with theGraphical User Interface (GUI). The upper left side of the GUI screengives centroid information for target circles in the current frame, andthe lower left side gives the same information for the prior frame. Foreach, one nested target circle from the set of three is displayed innegative (white on black) along with a histogram of its pixel brightnesswithin a 48-by-48 pixel box centered on the centroid location of theprevious frame. This histogram is split into two sections to display (atleft) the peak from background pixels at one end of the brightnessscale, and (at right) the peak from the pixels of the target itself, atthe other end of the brightness scale. A long continuum of pixels inbetween represents pixels at dark-light boundaries in the target frame.From analysis of the two histograms, the target field is clipped at thelower-brightness shoulder on the bright side, and the upper brightnessshoulder on the dark side, to create a binary target field that is notsensitive to variations in illumination across the target. Althoughdisplayed in real time for only one target circle, all three targetcircles are processed in this way.

Next to the target histograms, the x-y camera centroid locations aredisplayed to two-decimal precision for each of the three nested circletargets; again at the upper half of the screen for the current data andat the lower half of the screen for the prior frame.

The right side of the screen displays the processed 6-DOF data, afteranalysis using the approach described in Section 2.5. An analogmeter-style display shows the acquisition and processing time per frame,which is limited at its low end to the camera frame integration andreadout time of about 8 milliseconds. Using a single iteration of theNewton-Raphson routine described in Section 2.5, the algorithm runsduring the integration period for the successive frame, so theprocessing time is approximately 8 milliseconds, corresponding to a 120Hz camera readout rate. The 6-DOF data can be displayed in either analogor digital format, but the digital format can be read to precision of0.01 mm and 0.01 degree for comparison with the 0.1 mm, 0.1 degreeaccuracy requirements.

Laboratory Mechanical Layout for Head Tracking Simulation

The laboratory setup was designed to mimic head rotation anddisplacement using a six-degree-of-freedom optical rotation mount. Thismount included three ganged translation stages along the x-, y-, andz-axes of the optical table, and three ganged rotation stagescorresponding to yaw, roll and pitch respectively. The two monochromecameras and turning mirrors were mounted in the appropriate geometry foruse with an existing 12-channel head coil. The two monochrome camerasare in foreground, mounted at ±45° relative to horizontal to accommodaterotation by the turning mirrors. The turning mirrors are mounted 10inches behind cameras (slightly obscured by the cameras in the picture).The target is partially visible in the reflection of each mirror. The6-DOF rotation stage is at center in foreground, with the y-axis stageat bottom, x-axis stage next, and z-axis stage above that, followed bythe yaw rotation stage, the roll stage, and finally the pitch stage withtarget at the top (the pitch rotation handle is obscured by the stage).A near-IR illumination LED is at the center in background; light fromthis stage is within the camera spectral range, but hardly visible tothe human eye.

X-Axis Translation

The second translation stage from the bottom in the 6-DOF displacementassembly controls x-axis displacement (aligned with the patient'sspine). The x-axis translation stage control knob is turned four fullrotations (corresponding to −2.54 mm), and the absolute position changeis calculated from the resulting motion of the centroid cameracoordinates. Results are: the displacement determined by the unfoldingalgorithm is −2.56 mm in x, less than 0.1 mm in y and z, and less than0.1° in roll, pitch and yaw. The target displacement by dx=−2.54 mm,with zoom on lower right display section of GUI showed calculateddx=−2.56 mm, dy=0.08 mm, dz=0.02 mm, dφ=0.05°, dθ=−0.03°, and dψ=−0.01°.

Y-Axis Translation

The bottom translation stage in the 6-DOF displacement assembly controlsy-axis displacement (patient's left-to-right). The y-axis translationstage control knob is turned four full rotations (corresponding to −2.54mm), and the absolute position change is calculated from the resultingmotion of the centroid camera coordinates. This resulted in a targetdisplacement by dy=−2.54 mm, with zoom on lower right display section ofGUI showing dx=0.00 mm, dy=−2.47 mm, dz=−0.01 mm, dφ=0.64°, dθ=−0.04°,and dψ=−0.03°.

Z-Axis Translation

The top translation stage in the 6-DOF displacement assembly controlsz-axis displacement (patient's down to up, with the patient lying on hisback). The z-axis translation stage control knob is turned four fullrotations (corresponding to −2.54 cm), and the absolute position changeis calculated from the resulting motion of the centroid cameracoordinates. The displacement determined by the unfolding algorithm was−2.54 mm in z, less than 0.1 mm in x and y, and less than 0.1° in roll,pitch and yaw. The results were a target displacement by dz=−2.54 mm,with zoom on lower right display section of GUI showing dx=0.01 mm,dy=−0.01 mm, dz=−2.59 mm, dφ=−0.02°, dθ=−0.06° and dψ=0.01°.

Yaw Rotation

The bottom rotation stage in the 6-DOF displacement assembly controlsyaw rotation (patient's left shoulder—to—right shoulder lean direction).The yaw rotation stage control knob is turned by +4° degrees (heading315° to heading 311° on stage, corresponds to movement toward rightshoulder), and the absolute position change is calculated from theresulting motion of the centroid camera coordinates. The displacementdetermined by the unfolding algorithm is less than 0.1 mm in dx, dy anddz, 0.1° in roll and less than 0.1° in pitch, and 3.94° in yaw. Theresults were a target rotation by dψ=+4.00°, with zoom on lower rightdisplay section of GUI showing dx=0.07 mm, dy=−0.05 mm, dz=0.02 mm,dφ=0.10°, dθ=−0.01°, and dψ=3.94°.

Roll Rotation

The middle rotation stage in the 6-DOF displacement assembly controlsroll rotation (patient's right shoulde—to—left shoulder “head shaking”direction). The roll goniometer control knob is turned by +5° degrees,and the absolute position change is calculated from the resulting motionof the centroid camera coordinates. The displacement determined by theunfolding algorithm is less than 0.1 mm in dx, and dz, 1.78 mm in dy,4.97° in roll and less than 0.1° in pitch and yaw. Displacement in y isexpected due to the fact that the center of rotation for the ThorlabsGNL18 goniometer stage is 44.5 mm above the mount surface, while thetarget is only 22 mm above the stage. For the resulting −20.5 mm leverarm, the y-displacement due to a 5° roll rotation is −(−20.5mm)*sin(5°)=+1.79 mm, in good agreement with the measured data.

The results were a target rotation by dφ=+5.00°, with zoom on lowerright display section of GUI showing dx=0.07 mm, dy=1.78 mm, dz=−0.01mm, dφ=4.97°, dθ=−0.03°, and dψ=0.08°.

Pitch Rotation

The top rotation stage in the 6-DOF displacement assembly controls pitchrotation (patient's “nodding” direction). The pitch goniometer controlknob is turned by +5° degrees, and the absolute position change iscalculated from the resulting motion of the centroid camera coordinates.The calculated pitch is 4.95°, with less than 0.1° in yaw. The center ofrotation for the Thorlabs GNL10 goniometer stage is 25.4 mm above themount surface, while the target is only 6.4 mm above the stage. For theresulting −19 mm lever arm, the x-displacement due to a 5° rotation is−19 mm*sin(5°)=−1.66 mm, the y-displacement is 0.00 mm, and thez-displacement is −19 mm*[1−cos(5°)]=0.07 mm. These displacements areall within 0.1 mm of measured data.

The results were a target pitch rotation by dθ=+5.00°, with zoom onlower right display section of GUI showing dx=−1.63 mm, dy=0.09 mm,dz=0.17 mm, dφ=0.21°, dθ=4.95°, and dψ=−0.07°.

Laboratory Testing of an Embodiment

FIGS. 27A-27G illustrate a laboratory configuration testing anembodiment of the concepts described herein. FIG. 27A shows a top viewof a laboratory configuration comprising two optical detectors imagingan optical marker utilizing an approximately 90 degree scissor angle.FIGS. 27B-27G illustrate charts of motion tracking information ascalculated by an embodiment of a motion tracking system as a result ofmoving the optical marker to represent motion of an object beingtracked.

Additional Marker Embodiments

Various embodiments of optical trackers can be used with motion trackingsystems disclosed herein. For example, the marker 110 illustrated inFIG. 2A comprises a white or light colored background having three subtargets of a black or dark color. However, in some embodiments, it maybe advantageous to reverse this color scheme, for example, to use whiteconcentric sub targets on a black background. Such a marker can in someembodiments improve image processing and/or result in faster imageprocessing. As an example, in some embodiments, an image processingroutine monitoring the marker 110 illustrated in FIG. 2A includes aprocess of inverting a digital image of the marker so that the processeddigital image appears to be of a marker with a black background andwhite sub targets. To make the process more efficient, however, theimage inversion process can be eliminated when the marker itself alreadycomprises a dark background with light colored sub targets. The motiontracking process can accordingly exploit this contrast and a knowngeometry of the sub targets to rapidly locate centroids of the subtargets with minimal processing. FIG. 28 illustrates an embodiment of amarker 2810 comprising a black background and three white sub targets2804. Each of the sub targets 2804 comprises a plurality of concentricwhite ellipses, similar to as described above.

In some embodiments of markers, sub-targets comprise dark concentric subtargets on a high contrast diffusely reflective background. In someembodiments, the sub-targets comprise high contrast diffusely reflectiveconcentric sub-targets on a dark background. In some embodiments, theconcentric sub-targets comprise dark concentric sub-targets on a highcontrast retro-reflective background. In some embodiments, theconcentric sub-targets comprise high contrast retro-reflectiveconcentric sub-targets on a dark background. Retro reflective materialor surfaces are configured to reflect light back to a source withminimal scattering. Retro-reflective materials can provide benefits to amotion tracking system to, for example, in combination with theconcentric sub targets, enable even faster and efficient processing. Forexample, a background or sub target with retro-reflective properties canprovide additional information to the motion tracking system when themotion tracking system is analyzing an image of the target.

Various embodiments as disclosed herein have been disclosed withreference to a plurality of sub targets, in some embodiments three subtargets, on a single substrate that is connected to the patient beingmonitored. Further, in some embodiments, such as the embodimentsillustrated in FIGS. 6A and 6B, sub targets can be distributed, ratherthan located on a single substrate. As further examples, FIGS. 29A, 29B,and 30A through 30E illustrate additional embodiments of markers ortargets wherein individual sub targets are separated out onto their ownindividual substrates. For example, FIG. 29A illustrates a perspectiveview of three sub targets 2902 each on their own substrate 2904 attachedto a patient 112. In this embodiment, two of the substrates 2904 areattached to the patient's forehead, and one of the substrates 2904 isattached at the patient's mouth.

FIG. 29B illustrates another embodiment of a marker utilizingdistributed sub targets 2902 and substrates 2904. In this case, however,the three substrates 2904 are connected together and all attached to thepatient's mouth. The substrates 2004 may be connected together invarious ways, with some examples shown in FIGS. 30A through 30F.

FIG. 30A illustrates a front view of an embodiment of a distributedmarker comprising three sub targets 2902 each on their own substrate2904. The substrates 2904 are in turn connected to a backing 3006.Accordingly, although each sub target 2902 is on its own substrate 2904,the relative position of the three sub targets 2902 in relation to eachother remains the same.

FIGS. 30B through 30F illustrate a variety of embodiments of side viewsof a distributed marker utilizing a separate substrate 2904 for each subtarget 2902. The embodiment illustrated in FIG. 30B is a side view ofthe embodiment illustrated in FIG. 30A. In this embodiment, a mouthpiece3008, similar in design to the mouthpieces illustrated in FIGS. 5A-5D,is attached to the backing 3006 through a post 3010. Although in thisembodiment the backing is attached to the mouthpiece 3008 using a post3010, the backing 3006 may in some embodiments be directly attached tothe mouthpiece 3008 or attached to the mouthpiece using means other thana post. Further, although in these embodiments the markers areillustrated attached to a mouthpiece 3008 adapted to connect to a humanmouth, various embodiments of optical markers disclosed herein may beadapted to attached to other portions of an object being tracked. Forexample, a marker may be configured to attached to the bridge of a humannose, a chin, a forehead, one or more ears, hair, and/or the like. Insome embodiments, a marker attachment mechanism comprises a clamp sizedto engage a bridge of a nose. In some embodiments, a marker isconfigured to attach to an eyeglasses-like structure that engages boththe bridge of a person's nose and the person's ears.

FIG. 30C depicts an alternative embodiment of a side view of a markercomprising multiple substrates 2904. The embodiment illustrated in FIG.30C is similar to the embodiment illustrated in FIG. 30B. However, inthe embodiment illustrated in FIG. 30C, the substrates 2904 are eachconnected to the backing 3006 using their own post 3012.

FIG. 30D depicts an embodiment of a distributed marker wherein each ofthe substrates 2904 is directly attached to the mouthpiece 3008 usingits own post 3014. Accordingly, there is no backing 3006 connecting thesubstrates 2904 together. It should be noted that, although variousembodiments disclosed herein are described with reference to amouthpiece to attach the markers to a human mouth, markers may beattached in various other ways to a patient being monitored, as furtherdescribed above.

FIG. 30E depicts a marker with distributed substrates 2904 that issimilar in design to the marker depicted in FIG. 30D. However, themarkers illustrated in FIGS. 30E and 30F illustrate that in someembodiments the substrates 2904 and/or the sub targets 2902 do notnecessarily have to be in the same plane and/or oriented at the sameangle. In FIG. 30E, the substrates 2904 are oriented at differentangles. In FIG. 30F, the substrates 2904 are oriented at the same angle,but are not in the same plane. Such configurations may be advantageousin some embodiments, such as to enable viewing of markers and/or subtargets at greater ranges of motion. Further, markers or sub targets maybe in different planes to help conform a marker envelope shape to anarea available. For example, when an MRI head cage is around a patient,there may be little room in front of the patient's mouth. Accordingly,the substrates 2904 may need to be adjusted to fit within the head cage.In some embodiments, when a substrate 2904 is not in a same plane asanother substrate 2904, the printing of the sub targets 2902 may beadjusted so that to a camera the sub targets 2902 still appear as ifthey are in the same plane.

In various embodiments wherein sub targets 2902 are each positioned ontheir own substrate 2904, the substrates 2904 can be permanentlyattached to a backing 3006, removably attached to a backing 2006, orsetup in various other ways to be permanently attached together orremovably attached. For example, a substrate 2904 may be connected to abacking 3006 using glue, other adhesive, hook and loop fasteners,screws, tape, and/or the like.

Additional Process Flow Embodiments

FIG. 31 depicts an embodiment of a process flow diagram showing anexample of a motion compensation process. In this process flow, theprocess is analyzing movements of a marker 110 utilizing a whitebackground with black sub targets. However, a process such as theprocess flow shown in FIG. 31 may be modified to process a markerutilizing a dark background with light sub targets. An example of such aprocess may be seen below in FIG. 32.

The process flow begins at block 3102. At block 3102, the systeminitializes a video source object. At block 3104, the system grabs aframe from the video source, such as a camera. At block 3106, the systemanalyzes the frame to locate target centroids, such as centroids of thesub targets of a marker. The target centroid locating sub process isshown with reference to blocks 3108 through 3132. At block 3108, theprocess flow varies depending on whether this is the first iteration oflocating the target or whether it is a subsequent iteration. If it isthe first iteration, the process flow proceeds to block 3110. At block3110, the frame is converted to a binary image with a 50% threshold(e.g., any pixels with a value lower than 50% of maximum are set to zeroand any pixels with a value greater than 50% of maximum are set to one).At block 3112, the system identifies in the binary image the largestwhite region and selects that region. At block 3114, the system sets allother regions in the binary image to zero. At block 3116, the systeminverts the image.

At block 3118, the system again converts to a binary image with a 50%threshold. At block 3120, the system identifies the second, third, andfourth largest regions. At block 3122, the system sets all other regionsto zero. At block 3124, the system calculates centroids of the connectedregions. At block 3126, the system sorts and averages centroidcoordinates. The centroid coordinates are then passed on to theoperation at block 3134.

Returning to block 3108, if the process being performed at block 3106 isbeing performed subsequent to the first iteration, the process flowproceeds from block 3108 to block 3128. At block 3128, the systemextracts boxes around the last known centroid coordinates. At block3130, the system performs a weighted average to find a centroid in eachbox. At block 3132, the system adds a centroid offset to each of theprevious coordinates to produce the final centroid coordinates. Thesecoordinates are then passed on to the process at block 3134.

At block 3134, the system calculates the 3-D position and rotationangles of the optical target. In some embodiments, the 3-D position androtation angles are also converted from the optical detector coordinatesystem to, for example, an MRI machine coordinate system. At block 3136,the target position and rotation angles are passed onto an externalprocess to enable motion compensation.

FIG. 32 illustrates another embodiment of a process flow diagramillustrating an example of a centroid location process. For example, theprocess flow illustrated in FIG. 32 may be utilized to replace thesub-process illustrated in block 3106 of FIG. 31 when the target beingutilized is a target with a dark background and light sub targets.Accordingly, the process flow illustrated in FIG. 32 may enable a motiontracking system to perform faster and more efficiently than when using atarget with a light background and dark sub targets. FIG. 33 illustratesa variety of images depicting stages of the process flow illustrated inFIG. 32. Accordingly, some of the images in FIG. 33 will be referred toin describing the process flow of FIG. 32.

The process flow illustrated in FIG. 32 begins with a raw image of atarget from a camera. For example, block 3302 of FIG. 33 depicts anexample raw image from a camera of a target having a dark background andlight sub targets. At block 3202, the system converts the raw targetimage to a binary image with a threshold at less than 80% of the maximumpixel value. An example of a converted binary image can be seen in block3304 of FIG. 33. At block 3204, the system locates a target boundingbox. For example, in an embodiment, the systemic can be configured tofind the outermost nonzero rows and columns in the binary image. As canbe seen in block 3304 of FIG. 33, the target bounding box has beenadded.

At block 3206, the system is configured to identify nonzero pixels onthe bounding box perimeter. By identifying nonzero picture of pixels onthe bounding box perimeter, the system can estimate where each of thesub targets is. As can be seen in block 3304 of FIG. 33, the nonzeropixels on the bounding box perimeter have been identified and circled.For example, the points where the top left sub target touches thebounding box have been circled in red. The points where the top rightsub target touches the bounding box have been circling green. Finally,the point where the bottom sub target touches the bounding box perimeterhas been circled in blue. At block 3208, the system isolates each subtarget feature, for example, each concentric sub target, by assumingthat nonzero pixels on adjacent perimeter edges of a same quadrantbelong to the same target feature.

At block 3210, the system is configured to extract region of interest(ROI) boxes each containing one sub target. In some embodiments, theregion of interest box is size scaled by the size of the target boundingbox. As can be seen in block 3306 of FIG. 33, three region of interestboxes have been extracted from the binary image at block 3304.

At block 3212, the system is configured to set pixel values below 30% ofthe pixel maximum value to zero. For example, as seen in block 3308 ofFIG. 33, the concentric circles have been further isolated by settingpixel values below 30% of pixel maximum 20. At block 3214, the system isconfigured to locate the centroid of each sub target in each region ofinterest. For example, in some embodiments, the system can be configuredto take a maximum value of row and column sums of the products ofnormalized pixel values and pixel locations.

Block 3310 of FIG. 33 illustrates the sub target centroids calculated atblock 3214 overlaid on the original image from 3302. As can be seen, thesystem has efficiently determined the correct centroids of the subtargets. These centroids can then be passed on to other portions of themotion tracking process to enable tracking of a patient and compensatingfor that patients motion.

In some embodiments, a motion tracking system can be configured to runonly one iteration of the centroid determination process illustrated inFIG. 32 for each image frame being processed. By using only oneiteration, the system is efficient and quick in determining a patient'sposition. In some embodiments, however, the process flow illustrated inFIG. 32 or a similar process flow may comprise two or more iterations,such as 2, 3, 4, 5, 10, 15, 20, 30, 40, 50, or more iterations.

Additional Marker Embodiments

Various embodiments of optical trackers or markers used with motiontracking systems have been described in the present disclosure. Forexample, some embodiments, such as the marker 110 illustrated in FIG.2A, have comprised a white or light colored background having threesubtargets of a black or dark color, wherein the three subtargets arethree sets of concentric circles or ellipses located at the vertices ofan equilateral triangle. In other embodiments, the color scheme wasreversed such as in FIG. 28, so that the marker comprised a blackbackground with three white subtargets, wherein each subtarget was aplurality of concentric white ellipses. The subtargets in thisembodiment were also arranged so that the centroids of the subtargetswere equidistant from one another, such that they could be imagined asresiding on the vertices of an equilateral triangle.

However, in some embodiments, it may be advantageous to use a markerthat has no rotational ambiguity in order to allow for uniqueidentification in the location and orientation of the subtargets in themarker. A marker with three identically-sized subtargets, with theircentroids arranged as the vertices of an equilateral triangle, may belimited by rotational degeneracy. Such a marker may not have a uniquepattern or arrangement for every in-plane rotation angle (e.g.,rotations about an axis oriented normal to the plane of FIG. 2A,perpendicular to axes x and y of FIG. 2A). For example, rotating such amarker in-plane by 60 degrees would result in the same centroid patternas the marker before it was rotated, which may, in some embodiments,make it difficult to determine if there was even a rotation. In someembodiments, such as when the subtargets comprise concentric circles, anin-plane rotation of 60 degrees may result in both a same perceivedcentroid angular position and a same visual image of the marker. In someembodiments, such as when the subtargets comprise ellipses or othernonuniform shapes, an in-plane rotation of 60 degrees may result in adifferent visual image of the marker, even though the centroid angularposition may be perceived as the same as before the rotation. Someembodiments of tracking systems disclosed herein may be configured tocompensate for the potential rotational ambiguity of such a target, suchas by analyzing other differences in the visual image, and/or continuousor semi-continuous tracking of the marker during its in-plane rotationto gain additional context for analysis. It can be desirable, however,to utilize a marker that does not have such rotational ambiguity, whichmay allow faster and/or more efficient tracking. Thus, in someembodiments, the design of the marker may deviate from having threeidentically-sized, triangularly-arranged subtargets (or other designshaving in-plane rotational ambiguities) in order to avoid rotationaldegeneracy. There are many numerous, different patterns or designshaving a rotational asymmetry that would prevent rotational degeneracy.

FIG. 34 illustrates an example of a marker or optical tracker 3400 thatavoids rotational degeneracy, according to one embodiment. Furthermore,the marker 3400 in FIG. 34 may allow the system to locate the markerwith a high degree of accuracy regardless of the orientation of themarker in a sensor's field of view, and it may also allow the system tomore quickly identify the various subtargets of the marker. Marker 3400is shown to have five reference point locators, or subtargets, thatcomprise concentric pairs that are elliptical in shape and white coloredagainst a dark background. When these reference point locators areviewed from a 45 degree viewing angle by a detector, an image of thesereference point locators may illustrate the reference point locators ascircles. Marker 3400 may have four small concentric pairs 3402, whichare reference point locators located in the corners of marker 3400.Marker 3400 may also have a large concentric pair 3404, which is areference point locator positioned near the center of the marker (e.g.,subtarget 3404's centroid positioned within a bounding box defined bythe four centroids of the outer smaller subtargets 3402) that is largerin size than each of the small concentric pairs 3402. The largeconcentric pair 3404 may be offset in one direction of the marker, whichallows the relative positions of the five concentric pairs to uniquelyidentify the in-plane rotation of the marker. In this embodiment, thelarger subtarget 3404 is moved upward with respect to a verticallycentral portion or axis of the marker 3400 and/or a bounding box definedby the smaller subtargets 3402. In other embodiments, various othertechniques may be used to make the marker not rotationally ambiguous.For example, the larger central subtarget 3404 may be moved away from acenter point of the marker 3400 or bounding box defined by thesubtargets 3402 in a different direction. As another example, one ormore of the outer subtargets 3402 may be moved to a different location.

As mentioned above, in some embodiments, rotational asymmetry isachieved in the marker through an offset of one concentric target fromthe center of the marker along the vertical axis. This rotationalasymmetry may enable tracking of the marker through 360-degree in-planerotation with no degeneracy (e.g., no ambiguity as to the currentin-plane rotation orientation throughout a full 360 degrees). In someembodiments, the shapes of the subtargets of the marker are elongated inthe horizontal dimension to increase the horizontal cross section of themarker as it rotates away from a given camera. In some embodiments, themarkers may comprise three or more separate subtargets arranged in arotationally asymmetric pattern, with each subtarget comprising at leasttwo concentric shapes. Additional shapes may be incorporated into themarker pattern for the purposes of unique marker identification.

Since all five subtargets of the marker 3400 of FIG. 34 (the four smallconcentric pairs 3402 and the large concentric pair 3404) compriseconcentric shapes, the system is able to uniquely and/or efficientlyidentify and locate each of the five subtargets from an image of themarker. The system may do this in some embodiments by finding thecentroids of all the objects in a given size range, and then groupingobjects with the same (or sufficiently similar) centroid coordinates.All the objects with the same (or sufficiently similar) centroidcoordinates may be marked as a valid subtarget that is part of theoverall marker 3400. Utilizing such a design, where one or more subtargets comprise at least two concentric shapes, can be advantageous,for example, to enable quickly and/or efficiently identifying valid subtargets and ignoring other objects in an image of the marker. Forexample, when a marker as disclosed herein is utilized during an MRIbrain scan, a detector that captures an image of the marker may alsocapture other non-marker features in the image. For example, the imagecaptured by the detector may also include features of a patient's skincomplexion, such as moles or other items. When processing such an image,the system may identify a facial feature, such as a mole, as being afeature having a particular centroid location. However, if there is notanother feature having a same or similar centroid location, the systemcan confidently and efficiently filter out the mole as a potentialmarker sub target. Accordingly, it can be advantageous to utilize two ormore concentric shapes, or shapes having a different common geometricalindicator, to, among other things, efficiently filter out parts of animage that do not contain a valid subtarget. Although in thisembodiment, the subtargets each comprise a white dot and a white ringaround that white dot, various other embodiments of markers may comprisevarious other shapes. For example, two, three, four, five, or morecircles, ellipses, hexagons, octagons, and/or other shapes may beutilized. Further, within the same marker, different types of subtargetsmay be utilized. For example, one subtarget may comprise a differentnumber of concentric shapes than another subtarget, which may, forexample, provide context that enables tracking of in-plane rotationsthroughout the full 360 degrees even in a design where the subtargetcentroid positions alone are rotationally symmetrical (such as theequilateral triangle design of FIG. 2A).

Since large concentric pair 3404 is larger than the small concentricpairs 3402, the system may be able to more quickly identify largeconcentric pair 3402 than the other concentric pairs. The position ofthe large concentric pair 3404 may then be used to quickly locate theremaining concentric pairs—the four small concentric pairs 3402—througha variety of methods. One method for locating the remaining concentricpairs may be taking the four closest concentric pairs that surround thelarge concentric pair 3402, and then grouping all five concentric pairsas a marker. Thus, the large concentric pair 3402 may be used toidentify the location of the marker quickly. Otherwise, the centroids ofeach concentric pair can be located but the specific subtarget eachconcentric pair corresponds to is unknown. The concentric pairs may insome embodiments be put into local groups to determine the subtarget foreach concentric pair, such as through an algorithm like perimeter edgedetection, described previously herein. This may in some embodiments addundesired computational time and/or complexity to the algorithm foridentifying the location of the marker or uniquely identifyingsubtargets. Further details on marker pattern centroid detection areprovided below in the discussion associated with FIG. 37 and FIG. 38.

FIG. 35 illustrates another example of a marker or optical tracker thatavoids rotational degeneracy, according to one embodiment. In FIG. 35,there is an additional dot 3502 that is not a concentric pair.Additional shapes like dot 3502 may be added to the marker pattern forunique marker identification. In some embodiments of the system,multiple markers may be used simultaneously. The multiple markers mayhave related, rotationally asymmetric patterns with small variations,such as the number of additional shapes like dot 3502 that are includedin the pattern. The number, location, or arrangement of the shapes ordots may assist the system in unique marker identification, such thatthe system may be able to distinguish the markers from one another. Thesystem may separate the different markers from each other by searchingfor non-concentric shapes, marks, or objects such as dot 3502 in thebounding box region. The number of non-concentric shapes may be added upto determine which marker number it is. Further details on multi-markerdetection are provided below in the present disclosure. Although theembodiment illustrated in FIG. 35 utilizes a dot or circle 3502 as anadditional shape to enable marker unique identification and/or avoidanceof rotational degeneracy, various other embodiments of markers mayutilize other shapes for these purposes, such as two or more concentricshapes, a bar, a star, a square, a rectangle, and/or the like. It may bedesirable to have a set of markers that are individually uniquelyidentifiable, such as when multiple markers are used during a sametracking procedure. For example, during an MRI brain scan, the patientmay have two, three, four, five, six, or more markers simultaneouslyaffixed to his or her head, for example, to increase accuracy,redundancy, and or efficiency of the tracking process. In someembodiments, patterns are provided that uniquely identify an individualmarker with a given set of markers (e.g., a set of 2, 3, 4, 5, 6, 7, 8,9, 10, or more markers that are intended to be used simultaneously forthe same scan). In some embodiments, however, an individual marker maybe uniquely identifiable with respect to a larger set of markers or allother markers, not just markers with in a specific smaller subset ofmarkers that are intended to be used simultaneously. For example, amarker may comprise a bar code, QR code, and/or the like that can bedecoded to interpret a unique identifier. In some embodiments, such anencoded unique identifier may not necessarily be decoded and/or usedduring a motion tracking process, but may be saved associated withmotion tracking and/or scan data for later lookup and/or traceability.

FIGS. 36A and 36B each illustrate multiple embodiments of markers withvariations in the amount of non-concentric dots for the purposes ofunique marker identification and/or additional avoidance of rotationaldegeneracy. FIG. 36A illustrates multiple embodiments of markers thatpossess slight variations on the embodiment illustrated in FIG. 34. Theleftmost marker shown in FIG. 36A is identical to the embodimentillustrated in FIG. 34. There are four small concentric pairs 3602 inthe corners, with a large concentric pair 3604 near the center butslightly offset in the vertical direction. The marker immediately to theright has an additional dot 3606 at the bottom of the marker. Going fromleft to right in FIG. 36A, each marker possesses an additional dot overthe previous marker—starting with a marker having zero non-concentricdots to a marker having five non-concentric dots. Thus, FIG. 36A showssix markers with variations in the amount of non-concentric dots, whichmay all be used and uniquely identified simultaneously by the systembecause each marker is unique.

FIG. 36B also illustrates multiple embodiments of markers that comprisevariations. Each marker in FIG. 36B has five same-sized concentric pairs3608. Four of the concentric pairs are in the corners of each of themarkers, with the last concentric pair near the center of each markerbut possessing a vertical offset. Going from left to right in FIG. 36B,each marker possesses an additional non-concentric dot 3608 over theprevious marker. Thus, FIG. 36B also shows six markers that havevariations in the amount of non-concentric dots they display, and theymay all be used and uniquely identified simultaneously by the systembecause each marker is unique. However, the markers in FIG. 36B lack thelarge concentric pair 3604 that the markers in FIG. 36A possess. Thismay mean that the markers in FIG. 36B do not allow the system to performthe method of quickly identifying a large concentric pair from the otherconcentric pairs, and then using the position of the large concentricpair to locate the remaining concentric pairs in the marker. Using themarkers in FIG. 36B may work as well as the markers in FIG. 36A forreducing rotational degeneracy. However, using the markers in FIG. 36Bmay result in a longer time for identifying the markers and centroidsfrom images, as more calculations may be needed. However, the markers ofFIG. 36B comprise larger unique identification features (e.g., dots)than the markers of FIG. 36A, which may also be beneficial to the systemand the processing time required to locate and identify markers.

There may be conflicting requirements associated with the size anddesign of the actual marker used. Making the marker as small as possiblemay be beneficial for the patient's convenience, and/or to enable moremarkers to be used. However, the marker may need to be large enough thatthe system may be able to accurately determine the centroids of themarker from a pixelated image taken of the marker. Furthermore, theremay be an ideal spacing associated with the subtargets in a marker.Increased distances between subtargets in a marker may result inimproved tracking, since the tracking algorithm observes differencesassociated with the movement of the centroids of the subtargets. Thelarger the marker is, the larger this distance is for a given motion,and thus the better the tracking will be. Further discussion on the sizeof a marker is provided above in the present disclosure. In someembodiments, it can be desirable to utilize a marker comprising outersubtargets having centroid locations that define a bounding square of a14×14 millimeter size. In other embodiments, the bounding square (orrectangle) may comprise one or more sides having a length of 5, 6, 7, 8,9, 10, 11, 12, 13, 15, 16, 17, 18, 19, 20, or more millimeters.

Marker Pattern Centroid Detection

Since the motion tracking system tracks the relative positions ofmarkers, it may be useful for the system to use a process that canreliably identify the centroids of the subtargets (also referred to asreference points or concentric rings, herein) in a marker. It may alsobe useful to quickly and uniquely identify the subtargets that comprisea market. Various processes or methods may be used to identify thesubtargets and the centroids of the subtargets. The system may usedifferent processes or methods based on the marker patterns. Some markerdesigns may allow the system to use more efficient processes or methodswhen identifying the subtargets and the centroids of the subtargets.

FIG. 37 is a flow chart that illustrates a marker pattern centroiddetection process, according to one embodiment. This process may beapplied to each marker image obtained from a detector, camera, oroptical sensor (or in some embodiments only a subset of marker images,such as to increase efficiency). In some embodiments, 0 to 5 additionaldots are used on the markers in order to uniquely identify up to 6unique markers (although various other unique identification methods maybe used, as described above). However, there is no technical limitationon the number of markers a system may apply this centroid extractionalgorithm to. Parallel processing may be used to simultaneously performthe marker centroid extraction algorithm for all markers in the imagesreceived by all cameras present in the system. When paired with atracking algorithm that may scale to any number of targets, the numberof markers used may be limited only by space and computational resourceconstraints. Further discussion of multi-marker tracking is providedbelow.

At block 3702, the system may binarize each marker image received. Thesystem may set all pixels above a certain threshold to be white, whilesetting all pixels below that threshold to be black. The system wouldthen proceed to block 3704 to detect all connected regions within eachimage. The connected regions may be defined as all groups of whitepixels that are touching each other. However, in some embodiments, thesystem may be configured to detect connected regions of all groups ofblack pixels, such as when the marker comprises black subtargets on awhite background.

At block 3706, the system then filters out connected regions that aretoo small to be a marker component (such as those less than 4 pixels insome embodiments). At block 3708, the system may filter out connectedregions that are too large to be a marker component (such as thosegreater than 2000 pixels in some embodiments). The system may performblock 3706 and block 3708 in parallel, or in some embodiments, thesystem may perform those two blocks sequentially. The system may performblock 3706 first and then block 3708 second, or the system may performblock 3708 first and then block 3706 second.

At block 3710, the system may calculate the centroids of the remainingconnected regions. At block 3712, the system may then compare thecentroids of each connected region. Those regions with centroids within1 pixel of each other are identified as concentric subtarget pairs atblock 3714. In other embodiments, other thresholds may be used, such 0,2, 3, 4, 5, or more pixels. At block 3716, the system may then averagethe centroids of the regions in each concentric pair and record them asa valid marker subtarget in block 3716.

Alternatively, the system may utilize a somewhat different process ifthe marker has a large concentric ring, such as the marker embodimentillustrated in FIG. 34. Since the centermost subtarget is enlarged withrespect to the other marker subtargets, it may be more quickly isolatedbased on size. The overall marker location within the image may then beidentified efficiently. An algorithm may group the four subtargets orconcentric rings nearest to each central subtarget, in order to isolateand group all of the subtargets comprising each marker. In such anembodiment, the system would perform these modified processes in lieu ofblocks 3710, 3712, and 3714.

At block 3718, the system may then determine the general in-planerotation of each marker by calculating the angle of the line segmentconnecting the overall centroid of the group of subtargets comprisingthe marker to the central subtarget. With the rotation angle determined,a rotation transform is applied to the marker at block 3720 to remap itto an upright position.

At block 3722, the system may then sort the subtargets comprising themarker by their relative position to the central marker. At block 3724 asystem may count all of the connected regions within the bounding box ofeach marker to uniquely identify each observed marker.

FIG. 38 illustrates how an image of a rotationally asymmetric marker maybe processed in order to locate the centroids in the marker, forexample, using the process flow illustrated in FIG. 37.

Image 3802 is a raw greyscale image of a marker with rotationalasymmetry, such as the marker in FIG. 34. This image is similar to theimage of the marker that a detector or optical camera would take.

The system then binarizes image 3802 in order to obtain image 3804 forlocating all white, connected regions (e.g., block 3702 of FIG. 37). Tobinarize a grayscale image, a threshold for 50% of full scale is set.All pixels above this threshold are made white, and all pixels belowthis threshold are converted to black pixels. Connected regions are allremaining groups of white pixels that are touching either. Variousconnected regions can be seen in image 3804. The system then finds andevaluates all the connected regions based on their size. The regionsthat are way too small (such as a couple of pixels) may be noise ratherthan objects, and the system may discard those regions. The regions thatare too big (such as greater than 2000 pixels) may be large objects inthe image rather than components of the marker, and the system maydiscard those regions as well. These may be, for example, the processesperformed at blocks 3704-3708 of FIG. 37.

The system may then be left with image 3806, which shows all theconnected regions that have not been discarded, as they are within thedefined size range that the system is looking for. At this point, thesystem may then calculate the centroids of all remaining objects in theimage (e.g., block 3710 of FIG. 37). Objects with centroids occurringwithin 1 pixel of each other (i.e., concentric objects) may beidentified as concentric pairs or a valid subtarget (e.g., block 3714 ofFIG. 37). The centroids of each object in a concentric pair may beaveraged, in order to determine a centroid for the concentric pair. Theend result is image 3808, which illustrates the identified centroids3810 of all five of the concentric pairs present in the marker shown inthe original image 3802. These identified coordinates may be used withvarious algorithms disclosed herein to estimate the marker's position insix degrees of freedom, thus enabling an estimate of the patient'sposition in six degrees of freedom. As can be seen in the aboveexamples, using concentric rings may be one method of uniquelyidentifying the subtargets that are part of the marker. By usingconcentric rings for the subtargets, when objects in an image areidentified as having the same centroid coordinates, the probability isgreatly in favor of the two objects belonging to the same subtarget pairand to the marker.

Generalized Six-Degree-of-Freedom Tracking System Overview

This disclosure previously described some embodiments in which a MRIHead Tracker (or other motion tracking system) may take real-time inputfrom two 2D imaging sensors and analyze the data to determine and reportmotions in six degrees of freedom with minimal latency. Some of thoseembodiments are configured to detect and measure three centroidpositions on the marker and utilize those positions to determine theposition of the patient's head, by combining data from the two images ofthe marker to generate six unique principal quantities. One example ofsuch a process is illustrated in FIG. 20C. It can be desirable, however,to utilize a motion tracking system that can use any number of cameras,and/or any number of markers, including markers having various designs.Further, it can be desirable to have a motion tracking system that candynamically adjust to the addition and subtraction of one or morecameras and/or markers (and/or the blocking and unblocking of one ormore sight lines between cameras and markers). For example, as a patientmoves, one or more sightlines may become blocked, and it can bedesirable to have a motion tracking system that can dynamically andautomatically adapt to the sightlines that become blocked and/orunblocked during a medical imaging scan.

In some embodiments, a generalized six-degree-of-freedom algorithm isutilized that enables any non-zero number of cameras (and/or anynon-zero number of markers) to be used with the system. Such asix-degree-of-freedom tracking system may utilize 1 to n cameras with noparticular restrictions on camera locations, except that the field ofview of each camera covers some or the entire region of interest andthat the six-degree-of-freedom position of each camera is known in acommon arbitrary coordinate system. In some embodiments, an increase inthe number of cameras and/or markers increases the overall trackingaccuracy and/or redundancy. However, an increase in the number ofcameras and/or markers may also cause an increase in processing powerand/or processing time required to estimate a patient's pose.Accordingly, it can be desirable to optimize the number of camerasand/or number of markers used based on the available processing power,the available space in, on, or around the medical scanner forpositioning therein of cameras, and/or the available space and/ormounting positions for markers on the patient being scanned. In someembodiments, such as in an MRI scanner where a patient is scanned withina generally cylindrical bore, the system can be configured to position aplurality of cameras about the bore, with the cameras positioned suchthat at least one camera (or any other number of cameras) can always seeat least one marker (or any other number of markers) in any anticipatedposition of the patient during the scan.

In some embodiments, the system may use data from every camera that cancurrently see a marker (which may or may not include every camera in thesystem). For example, if cameras 1, 3, and 5 can see a marker, then thesystem may use data from just those three cameras while ignoring datafrom other cameras. The system may automatically determine the positionof the marker based on which cameras can see the marker. In someembodiments, the system may be able to seamlessly deal with blocking orobstructing of the viewpath of one or more cameras. For example, ifthere are initially three cameras that can see a marker but one camera'sviewpath becomes blocked, such as by the patient's arm, then the systemmay be able to use the other two cameras with unobstructed viewpaths todetermine the marker (or markers) position. In some embodiments, morethan one marker may be tracked by some or all of the cameras ordetectors.

In some embodiments, the accuracy of the motion tracking system may belower when one camera is used as compared to when more cameras are used,although the accuracy may still be sufficient to reduce or eliminatemotion artifacts in the medical imaging scans. The accuracy may improvewith two or more cameras, and the tracking accuracy may generallyincrease with additional numbers of cameras that can detect a marker(the same marker and/or other markers) at a given time. The generalizedsix-degree-of-freedom algorithm may allow the system to switchdynamically from camera to camera, or add and remove cameras from use.

In some embodiments, the system utilizes a generalized set of principalquantities (e.g., the principal quantities may be used with variousmarker designs). The principal quantities may comprise in someembodiments the sums and differences in the X and the Y coordinatesbetween every pair of targets from every image. For example, a markermay be imaged by both camera 1 and camera 2. The principal quantitieswould be the sum and difference in the X coordinate and the Y coordinatebetween every possible combination of pairs of targets available for agiven marker. The system may construct a set of principle quantities forany number of markers based on the data available. For example, if thereare six cameras being utilized by the system and three of the camerascan see the centroids of that marker, then the system may generate afull set of principal quantities based on all of that combined data. Inthis manner, the system may adjust the principal quantities based on newdata available, such as if a fourth camera was able to see the centroidsin the marker (or if only one camera were able to see a marker).Additional details regarding principal quantity calculations are givenbelow.

In some embodiments, the system may utilize six cameras. In someembodiments, the system may utilize four cameras located on thecircumference of a circle on or about the body coil of an MR scanner,pointed inward with positions adjusted for best coverage of theregion(s) of interest. The number of cameras used in the system may bechosen based on available processing power, space, optimum view paths,and/or the like. Additional cameras may improve accuracy and allow forredundancy, although in some embodiments additional cameras may resultin additional processing power being needed for a desired processingtime.

In some embodiments, the system may need to know the exact position andorientation of all the cameras in the system (or at least a position andorientation within sufficient tolerances). This information may beobtained in some embodiments as part of a separate calibration process,an example of which is further discussed below. The system may also needto know the relative positions of all the targets or subtargets within amarker. The relative positions of all the targets within a marker may insome embodiments be known before-hand based on the design of the marker.

Target Displacement Equations

In some embodiments, the marker nominal centroid positions aredetermined from the geometrical characteristics of the marker. The fullsix-degree-of-freedom translation is composed of a 3-D displacement aswell as a 3-axis rotation. The rotations are separated into roll, pitchand yaw (although other conventions may be used), and the translatedposition through rotation follows the Euler rotation matrix formulationas follows (using right-handed angle conventions). The x, y, and zdisplacement coordinates then follow the obvious independenttranslations:

$\begin{pmatrix}x^{\prime} \\y^{\prime} \\z^{\prime}\end{pmatrix} = {{\begin{pmatrix}1 & 0 & 0 \\0 & {\cos \; \phi} & {{- \sin}\; \phi} \\0 & {\sin \; \phi} & {\cos \; \phi}\end{pmatrix}\begin{pmatrix}{\cos \; \theta} & 0 & {\sin \; \theta} \\0 & 1 & 0 \\{{- \sin}\; \theta} & 0 & {\cos \; \theta}\end{pmatrix}\begin{pmatrix}{\cos \; \psi} & {{- \sin}\; \psi} & 0 \\{\sin \; \psi} & {\cos \; \psi} & 0 \\0 & 0 & 1\end{pmatrix}\begin{pmatrix}x \\y \\z\end{pmatrix}} + \begin{pmatrix}{\Delta \; x} \\{\Delta \; y} \\{\Delta \; z}\end{pmatrix}}$

Decomposing the six independent translations from the absolute andrelative displacements of the measured marker centroids is the subjectof this effort. The 2D inverse problem is somewhat more difficult thanthe 3D problem, in that after the marker centroid projections in focalplane row and column are determined, significant degeneracies remain inthe unfolding matrices for each camera. Combining the data from multiplecameras removes these degeneracies through a series of interrelated,nonlinear equations. The fastest procedure for solving this inverseproblem is obtained by a variant of the Newton-Raphson method, wherebyan approximate first-order solution is proposed and tested against theknown (measured) centroid locations on the camera focal planes. Theresidual error is divided by the local derivatives with respect to eachcomponent of rotation and translation, to determine an iterativecorrection. The first-order solution is chosen by considering thefeatures of the projected target pattern which are most stronglyaffected by a single rotation angle or displacement, and linearizing theinversion problem along these feature axes.

Six-Degree-of-Freedom Extraction Algorithm Image Characterization andCalculation of Principal Quantities

In this embodiment, the markers each comprise 3 or more targets (plusadditional identifying marks in some embodiments) and may be observed by0 to n cameras, where n is the total number of active cameras in thesystem. In the case of a marker being obscured or moved outside of thefield of view of all cameras, its position is unknown and not consideredin any subsequent calculations until such time as the marker moves to aposition where is can again be detected. In other cases,characterization of the images is achieved, in this embodiment, bycalculating both the sum and difference of the column (X) and row (Y)coordinates between each target image and every other target imageacross all cameras. For a given marker, with targets, T and cameras, C,the set of principal quantities (PQ) characterizing each marker is givenby:

PQ _(x) ={X _(T) _(i) _(C) _(k) ±X _(T) _(j) _(C) _(i) |i,jε[1:m]

k,lε[1:n]

C _(k) C _(l) contain valid marker images}

PQ _(y) ={Y _(T) _(i) _(C) _(k) ±Y _(T) _(j) _(C) _(i) |i,jε[1:m]

k,lε[1:n]

C _(k) C _(l) contain valid marker images}

PQ=PQ _(x) ∪PQ _(y)

In this manner, a set of principal quantities, PQ, may be constructedfor each marker based on all available image data.

Characterizing Global Variation in Principal Quantities withSix-Degree-of-Freedom Motions

Partial derivatives relative to subject displacements and rotations (φ,θ, ψ, Δx, Δy, Δz), of the principal quantities described above, aboutthe initial (non-displaced) position, are computed numerically. Here:

Roll φ is right-handed rotation about the x-axisPitch θ is right-handed rotation about the y-axisYaw ψ is right-handed rotation about the z-axisΔx is toe-to-head directionΔy is left-to-right directionΔz is down-to-up direction

Starting from an initial marker position in 3-D world space, defined as(φ, θ, ψ, Δx, Δy, Δz)=(0, 0, 0, 0, 0, 0), the initial world coordinates(x_(0i),y_(0i),z_(0i)) are determined for each target, i, comprising themarker based on the known geometric configuration of the marker anddefinition of a convenient coordinate origin.

Local partial derivatives of each of the principal quantities, withrespect to each of the 6 degrees of freedom (roll, pitch, yaw, dx, dy,dz), are performed numerically by evaluating changes in these quantitiesfor small increments in each degree of freedom. Changes in the targetpositions for specified motions along the six degrees of freedom arecomputed using the Euler rotation matrices and translation vector:

$\begin{matrix}{\begin{pmatrix}x_{i} \\y_{i} \\z_{i}\end{pmatrix} = {{\begin{pmatrix}1 & 0 & 0 \\0 & {\cos \; \phi} & {{- \sin}\; \phi} \\0 & {\sin \; \phi} & {\cos \; \phi}\end{pmatrix}\begin{pmatrix}{\cos \; \theta} & 0 & {\sin \; \theta} \\0 & 1 & 0 \\{{- \sin}\; \theta} & 0 & {\cos \; \theta}\end{pmatrix}\begin{pmatrix}{\cos \; \psi} & {{- \sin}\; \psi} & 0 \\{\sin \; \psi} & {\cos \; \psi} & 0 \\0 & 0 & 1\end{pmatrix}\begin{pmatrix}x_{0\; i} \\y_{0\; i} \\z_{0\; i}\end{pmatrix}} + \begin{pmatrix}{\Delta \; x} \\{\Delta \; y} \\{\Delta \; z}\end{pmatrix}}} & \left\lbrack {1A} \right\rbrack\end{matrix}$

Subsequently, the normalized pinhole camera model projections of the newmarker target positions are determined using a geometric projectioncalculation. Given accurate knowledge of camera positions, thenormalized projections are given by:

x _(i,j)=

tan α_(i,j) sin β_(i,j) ,y _(i,j)=±tan α_(i,j) sin β_(i,j)  [2A]

Here x and y_(i,j) are the horizontal and vertical pinhole cameraprojections for translated target i projected onto the camera j sensor,where the angles α_(i,j) and β_(i,j) are the polar and azimuth angleslocating target i, relative to the camera j focal axis. In order tomaintain the convention of a counterclockwise sense of rotation for theazimuth angle for each of the camera boresight axes, equation 2 isdivided into two mirrored functions: one for cameras to the patient'sleft side and the second for cameras to the patient's right side. Thepolar and azimuth angles are calculated from the target position worldcoordinates as follows:

$\begin{matrix}{{\alpha_{i,j} = {\sin^{- 1}\left( \frac{\sqrt{\left( {x_{\bot{,j}} - x_{i}} \right)^{2} + \left( {y_{\bot{,j}} - y_{i}} \right)^{2} + \left( {z_{\bot{,j}} - z_{i}} \right)^{2}}}{\sqrt{\left( {x_{cj} - x_{i}} \right)^{2} + \left( {y_{cj} - y_{i}} \right)^{2} + \left( {z_{cj} - z_{i}} \right)^{2}}} \right)}},} & \left\lbrack {3A} \right\rbrack \\{\beta_{i,j} = {\pm {{\cos^{- 1}\left( \frac{{\left( {x_{\bot{,j}} - x_{i}} \right)\left( {y_{cj} - y_{0}} \right)} - {\left( {y_{\bot{,j}} - y_{i}} \right)\left( {x_{cj} - x_{0}} \right)}}{\begin{matrix}\sqrt{\left( {x_{\bot{,j}} - x_{i}} \right)^{2} + \left( {y_{\bot{,j}} - y_{i}} \right)^{2} + \left( {z_{\bot{,j}} - z_{i}} \right)^{2}} \\\sqrt{\left( {x_{cj} - x_{0}} \right)^{2} + \left( {y_{cj} - y_{0}} \right)^{2}}\end{matrix}} \right)}.}}} & \left\lbrack {4A} \right\rbrack\end{matrix}$

Here the point (x_(⊥,j),y_(⊥,j),z_(⊥,j)) is the point of intersectionbetween the camera optical axis and the plane perpendicular to theoptical axis which includes the translated target position(x_(i),y_(i),z_(i)):

x _(⊥,j) =x ₀+κ(x _(cj) −x ₀);y _(⊥,j) =y ₀+κ(y _(cj) −y ₀);z _(⊥,j) =z₀+κ(z _(cj) −z ₀),  [5A]

with (x_(cj),y_(cj),z_(cj)) defining the 3-D position of camera j,(x₀,y₀,z₀) defining the nominal boresight position of all cameras at theun-displaced target center and the constant based on geometricprojection and given by:

$\begin{matrix}{\kappa = {\left\{ \frac{{\left( {x_{cj} - x_{0}} \right)\left( {x_{i} - x_{0}} \right)} + {\left( {y_{cj} - y_{0}} \right)\left( {y_{i} - y_{0}} \right)} + {\left( {z_{cj} - z_{0}} \right)\left( {z_{i} - z_{0}} \right)}}{\left( {x_{cj} - x_{0}} \right)^{2} + \left( {y_{cj} - y_{0}} \right)^{2} + \left( {z_{cj} - z_{0}} \right)^{2}} \right\}.}} & \left\lbrack {6A} \right\rbrack\end{matrix}$

In equation [4], the inverse cosine function is taken to range from 0 toπ, and the appropriate sign for β_(i,j) is given by:

[(z _(cj) −z ₀){(x _(cj) −x ₀)(x _(⊥,j) −x _(i))+(y _(cj) −y ₀)(y _(⊥,j)−y _(i))}−(z _(⊥,j) −z _(i)){(x _(cj) −x ₀)²+(y _(cj) −y ₀)²}]

To numerically evaluate the partial derivatives of the principalquantities about the initialized target position, the un-displaced 3-Dtarget position coordinates (x_(0i),y_(0i),z_(0i)) are first projectedto camera coordinates using equations [2A] through [6A] above, andinitial values are computed for each of the principal quantitiesdescribed above (most should be zero or near-zero at the startingposition). Then small increments of roll, pitch, yaw, x-, y- and z-axisdisplacements are introduced one at a time; for each increment the newworld coordinates and the new camera projections of the target positionsare computed and the principal quantities are re-calculated. The changein each principal quantity is divided by the small angular ordisplacement increment to determine the partial derivative.

Radial and tangential lens distortion effects are applied to thenormalized projections according to the following equation:

$\begin{bmatrix}x_{i,{j\mspace{14mu} {distorted}}} \\y_{i,{j\mspace{14mu} {distorted}}}\end{bmatrix} = {{\left( {1 + {ar}^{2} + {br}^{4} + {cr}^{6}} \right)\begin{bmatrix}x_{i,j} \\y_{i,j}\end{bmatrix}} + \begin{bmatrix}{{2\; {dxy}} + {e\left( {r^{2} + {2x^{2}}} \right)}} \\{{d\left( {r^{2} + {2\; y^{2}}} \right)} + {2\; {exy}}}\end{bmatrix}}$

Where r²=x²+y², and the radial distortion coefficients, a, b, c and thetangential distortion coefficients d, e, are determined by a separatelens calibration procedure. Lastly, the distorted projection coordinatesare mapped to pixel values for each camera as follows:

${X_{i,j} = {X_{0} + {\left( \frac{f.l.}{s_{pix}} \right)x_{i,{j\mspace{14mu} {distorted}}}}}},{Y_{i,j} = {Y_{0} + {\left( \frac{f.l.}{s_{pix}} \right)y_{i,{j\mspace{14mu} {distorted}}}}}}$

Where f.l. and s_(pix) are the lens focal length and camera pixel pitch,and X₀ and Y₀ are the pixel coordinates of the center of the pixel array(i.e. for a 1280×1024 array, X₀=640.5 and Y₀=512.5, assuming the centerof the upper left corner pixel is defined as having coordinates (1,1).).

For instance, to determine the partial derivatives with respect to roll,the displacement vector (φ, θ, ψ, Δx, Δy, Δz)=(ôφ, 0, 0, 0, 0, 0) isintroduced to the general displacement equation [1] to determine thetranslated target positions (x_(i),y_(i),z_(i)): The conversion tocamera coordinates (X_(i,j), Y_(i,j)) is then performed using equations[2] through [6], and the principal quantities are calculated as outlinedabove. The difference between each principal quantity and thecorresponding value of that quantity for the un-displaced calculation isdivided by the small increment in roll, to give the partial derivativeof each quantity with respect to roll. To determine partial derivativeswith respect to pitch, the displacement vector (φ, θ, ψ, Δx, Δy, Δz)=(0,ôθ, 0, 0, 0, 0) is used to initiate the calculations, and so on for allsix degrees of freedom.

Each of these six repetitions produces one column of the global partialderivative matrix:

${\begin{pmatrix}\frac{\partial{PQ}_{1}}{\partial\phi} & \frac{\partial{PQ}_{1}}{\partial\theta} & \frac{\partial{PQ}_{1}}{\partial\psi} & \frac{\partial{PQ}_{1}}{\partial x} & \frac{\partial{PQ}_{1}}{\partial y} & \frac{\partial{PQ}_{1}}{\partial z} \\\vdots & \vdots & \vdots & \vdots & \vdots & \vdots \\\frac{\partial{PQ}_{t}}{\partial\phi} & \frac{\partial{PQ}_{t}}{\partial\theta} & \frac{\partial{PQ}_{t}}{\partial\psi} & \frac{\partial{PQ}_{t}}{\partial x} & \frac{\partial{PQ}_{t}}{\partial y} & \frac{\partial{PQ}_{t}}{\partial z} \\\vdots & \vdots & \vdots & \vdots & \vdots & \vdots \\\frac{\partial{PQ}_{n}}{\partial\phi} & \frac{\partial{PQ}_{n}}{\partial\theta} & \frac{\partial{PQ}_{n}}{\partial\psi} & \frac{\partial{PQ}_{n}}{\partial x} & \frac{\partial{PQ}_{n}}{\partial y} & \frac{\partial{PQ}_{n}}{\partial z}\end{pmatrix}}_{({0,0,0,0,0,0})}$

Determining First-Order Displacement Vector

A first-order approximation to the displacement matrix is determined bymultiplying the matrix of measured principal quantities, as determinedabove, by the inverse of the partial derivative matrix computed above:

$\begin{pmatrix}\phi_{0} \\\theta_{0} \\\psi_{0} \\\left( {\Delta \; x} \right)_{0} \\\left( {\Delta \; y} \right)_{0} \\\left( {\Delta \; z} \right)_{0}\end{pmatrix} = {\begin{pmatrix}\frac{\partial{PQ}_{1}}{\partial\phi} & \frac{\partial{PQ}_{1}}{\partial\theta} & \frac{\partial{PQ}_{1}}{\partial\psi} & \frac{\partial{PQ}_{1}}{\partial x} & \frac{\partial{PQ}_{1}}{\partial y} & \frac{\partial{PQ}_{1}}{\partial z} \\\vdots & \vdots & \vdots & \vdots & \vdots & \vdots \\\frac{\partial{PQ}_{t}}{\partial\phi} & \frac{\partial{PQ}_{t}}{\partial\theta} & \frac{\partial{PQ}_{t}}{\partial\psi} & \frac{\partial{PQ}_{t}}{\partial x} & \frac{\partial{PQ}_{t}}{\partial y} & \frac{\partial{PQ}_{t}}{\partial z} \\\vdots & \vdots & \vdots & \vdots & \vdots & \vdots \\\frac{\partial{PQ}_{n}}{\partial\phi} & \frac{\partial{PQ}_{n}}{\partial\theta} & \frac{\partial{PQ}_{n}}{\partial\psi} & \frac{\partial{PQ}_{n}}{\partial x} & \frac{\partial{PQ}_{n}}{\partial y} & \frac{\partial{PQ}_{n}}{\partial z}\end{pmatrix}^{- 1}\begin{pmatrix}{PQ}_{1} \\\vdots \\{PQ}_{t} \\\vdots \\{PQ}_{n}\end{pmatrix}}$

Characterizing Local Variation in Principal Quantities withSix-Degree-of-Freedom Motions

First order values for (φ, θ, ψ, Δx, Δy, Δz) determined above areentered into the translation equation [1A] to determine thecorresponding translated 3-D target position (x_(i),y_(i),z_(i)) foreach of the target positions. These world coordinates are projected tocamera coordinates (X_(i,j), Y_(i,j)) using equations [2A] through [6A],and the principal quantities are re-calculated. These quantities arecompared against the measured values of these quantities determinedabove, to create a residual error matrix:

(σ_(PQ) ₁ , . . . ,σ_(PQ) _(i) , . . . ,σ_(PQ) _(n) )

Local partial derivatives of the principal quantities are calculated byintroducing small increments in roll, pitch, yaw, x-, y- and z-axisdisplacements one at a time as before, but this time the increments arerelative to the first-order displacement vector. For each increment, thenew world coordinates and the new camera projections of the targetpositions are re-computed and the principal quantities arere-calculated. The change in each principal quantity is divided by thesmall angular or displacement increment to determine a local partialderivative. For instance, to calculate partial derivatives with respectto roll, the first-order displacement vector {φ₀, θ₀, ψ₀, (Δx)₀, (Δy)₀,(Δz)₀} is replaced by {φ₀+ôφ, θ₀, ψ₀, (Δx)₀, (Δy)₀, (Δz)₀} and resultingchanges to each of the principal quantities is divided by to determinethe local derivative with respect to roll. This is repeated for each ofthe six degrees of freedom.

Each of these six repetitions produces one column of the new localpartial derivative matrix:

${\begin{pmatrix}\frac{\partial{PQ}_{1}}{\partial\phi} & \frac{\partial{PQ}_{1}}{\partial\theta} & \frac{\partial{PQ}_{1}}{\partial\psi} & \frac{\partial{PQ}_{1}}{\partial x} & \frac{\partial{PQ}_{1}}{\partial y} & \frac{\partial{PQ}_{1}}{\partial z} \\\vdots & \vdots & \vdots & \vdots & \vdots & \vdots \\\frac{\partial{PQ}_{i}}{\partial\phi} & \frac{\partial{PQ}_{i}}{\partial\theta} & \frac{\partial{PQ}_{i}}{\partial\psi} & \frac{\partial{PQ}_{i}}{\partial x} & \frac{\partial{PQ}_{i}}{\partial y} & \frac{\partial{PQ}_{i}}{\partial z} \\\vdots & \vdots & \vdots & \vdots & \vdots & \vdots \\\frac{\partial{PQ}_{n}}{\partial\phi} & \frac{\partial{PQ}_{n}}{\partial\theta} & \frac{\partial{PQ}_{n}}{\partial\psi} & \frac{\partial{PQ}_{n}}{\partial x} & \frac{\partial{PQ}_{n}}{\partial y} & \frac{\partial{PQ}_{n}}{\partial z}\end{pmatrix}}_{\{{\phi_{0},\theta_{0},\psi_{0},{({\Delta \; x})}_{0},{({\Delta \; y})}_{0},{({\Delta \; z})}_{0}}\}}$

Determining Coarse Correction to the First-Order Displacement Vector

A coarse correction is computed to improve the first-order displacementvector and reduce residual error, by multiplying the residual errormatrix determined above by the inverse of the local partial derivativematrix, also determined above:

$\begin{pmatrix}{\Delta \; \phi} \\{\Delta \; \theta} \\{\Delta \; \psi} \\{\Delta \left( {\Delta \; x} \right)} \\{\Delta \; \left( {\Delta \; y} \right)} \\{\Delta \; \left( {\Delta \; z} \right)}\end{pmatrix} = {\begin{pmatrix}\frac{\partial{PQ}_{1}}{\partial\phi} & \frac{\partial{PQ}_{1}}{\partial\theta} & \frac{\partial{PQ}_{1}}{\partial\psi} & \frac{\partial{PQ}_{1}}{\partial x} & \frac{\partial{PQ}_{1}}{\partial y} & \frac{\partial{PQ}_{1}}{\partial z} \\\vdots & \vdots & \vdots & \vdots & \vdots & \vdots \\\frac{\partial{PQ}_{t}}{\partial\phi} & \frac{\partial{PQ}_{t}}{\partial\theta} & \frac{\partial{PQ}_{t}}{\partial\psi} & \frac{\partial{PQ}_{t}}{\partial x} & \frac{\partial{PQ}_{t}}{\partial y} & \frac{\partial{PQ}_{t}}{\partial z} \\\vdots & \vdots & \vdots & \vdots & \vdots & \vdots \\\frac{\partial{PQ}_{n}}{\partial\phi} & \frac{\partial{PQ}_{n}}{\partial\theta} & \frac{\partial{PQ}_{n}}{\partial\psi} & \frac{\partial{PQ}_{n}}{\partial x} & \frac{\partial{PQ}_{n}}{\partial y} & \frac{\partial{PQ}_{n}}{\partial z}\end{pmatrix}^{- 1}\begin{pmatrix}\sigma_{{PQ}_{1}} \\\vdots \\\sigma_{{PQ}_{t}} \\\vdots \\\sigma_{{PQ}_{n}}\end{pmatrix}}$

The first-order displacement vector is incremented by the coarsecorrection matrix to create a better approximation to the displacementvector:

{φ₀+Δφ,θ₀+Δθ,ψ₀+Δψ,(Δx)₀+Δ(Δx),(Δy)₀+Δ(Δy),(Δz)₀+Δ(Δz)}

Iterative Fine Correction to Determine Final Six-Degree-of-FreedomDisplacement Vector

Characterizing Local Variation in Principal Quantities withSix-Degree-of-Freedom Motions and Determining Coarse Correction to theFirst-Order Displacement Vector are repeated, starting with thecoarse-corrected displacement vector, to determine a fine correction tothe displacement vector. After this iteration, the resultant finecorrection increments are added to the coarse-corrected vector to createthe new 6-DOF displacement vector. These processes are repeatediteratively until the magnitude of each of the fine corrections is lessthan the desired threshold (typically 10⁻⁶ mm or degrees) or the numberof iterations reaches a predetermined maximum (typically 10 iterations).

One of skill in the art will recognize that the above-described sixdegree of freedom tracking algorithm is merely one way of implementingthe tracking techniques disclosed herein, and various modifications maybe made to this process without departing from the techniques disclosedherein.

Multi-Marker Tracking

Some embodiments of the motion tracking systems disclosed herein maysimultaneously use multiple markers, such as to increase accuracy and/orredundancy. It may be useful for the system to be able to distinguisheach marker from one another. One way of physically distinguishingmarkers from one another is described above, in which rotationallyasymmetrical markers have varying amounts of non-concentric dots orother indicators on them. Although these markers are physicallydistinct, it may be desirable for the system to distinguish and uniquelyidentify the markers algorithmically. This algorithm may be generalized,such that the system may simultaneously handle any number of markers,including a greater amount of markers than the arrangements shown inFIG. 36A and FIG. 36B which depict six varying markers. With thismulti-marker tracking algorithm, calculations may be performedseparately and completely independent for each marker in parallel. Somemarkers may be simultaneously tracked by more than one camera. Somemarkers may be tracked by whichever camera is able to see or detect themarker. Further discussion of an example algorithm performed for eachmarker is described below in relation to the processes outlined for thesix-degree-of-freedom extraction algorithm.

In addition to accuracy and redundancy benefits, the tracking ofmultiple markers may be helpful in determining the presence of non-rigidfacial movement. A marker may undergo rigid motion and move with theskeletal structure of the patient, or the marker may undergo non-rigidmotion that is not indicative of structural motion of the patient's heador other feature being tracked (e.g., from relative movement of apatient's skin, cartilage, and/or the like that is not rigidly connectedto the patient's skeletal structure). It may be helpful to distinguishwhat kind of motion a marker is undergoing, because non-rigid motion maylead to an erroneous correction by the motion tracking system if it isdetected as rigid motion.

Some embodiments of systems disclosed herein may keep track of a historyof the relative positions of all the markers in the system. Once thesix-degree-of-freedom coordinates for all visible markers aredetermined, coordinate transforms may be applied to remap the coordinatesystem origin sequentially to each of the markers, recording theresulting relative positions of the remaining markers in each case. Inthis manner, the six-degree-of-freedom relative position of each markerwith respect to every other marker may be determined. A history of therelative position data over several seconds is maintained and may beused to calculate the means and standard deviations of relativepositions of all markers. Each new set of relative position data may becompared to the historical mean and standard deviation, σ. If the newvalue is more than ±3σ from the historical mean (or another value inother embodiments), then it may be flagged as a non-rigid motion.

With the use of only one marker, the marker may be subject to non-rigidmotion but the system may in some embodiments be unable to determine it.

With two markers, the system may be able to look at the relativepositions in space of the two markers. The system may be able to comparethe latest set of coordinates for the two markers against their history.If the two markers move more than 3 sigma than their historical relativeposition (or another value in other embodiments), then the system maydetermine that there is non-rigid motion. However, with two markers itmay be difficult to tell if one of them (or both of them) are undergoingnon-rigid motion, only that it is happening.

With three or more visible markers, the system may be able to look atrelative positions of each marker with respect to the other markers, andcompare their history. The system may be able to determine that marker 2is moving relative to other markers, but all the other markers aremaintaining their relative positions to each other. This may mean thatmarker 2 is likely to be the non-rigid moving marker. The system maythen focus on the other markers that are moving rigidly, and use datafrom those markers in order to enhance accuracy. As another example, ifthere are three visible markers and non-rigid motion flags are onlyraised in the relative positions between markers 1&2 and markers 2&3,then it is deduced that either marker 2 is moving non-rigidly or markers1 &3 are moving non-rigidly with exactly the same motion (to within 3σ).Additional visible markers in the system would further reduce thealready unlikely probability of the latter.

The multi-marker tracking algorithm may also allow tracking to continuein situations when camera viewpaths are obscured, or objects move out ofthe field of view of the cameras. More specifically, if two or moremarkers are determined to be moving rigidly, then their position datamay be combined to further improve tracking accuracy. This may be doneby specifying one marker in the system as the ‘primary’ marker (this maybe done automatically, for example, by assigning the lowest numberedmarker in the system as the primary marker), whose six-degree-of-freedomposition is reported, and mapping all of the other visible, rigidlymoving markers to its position using the historical relative positiondata. Once mapped to the same coordinate position, thesix-degree-of-freedom coordinates are averaged. This method has theadditional benefit that if the primary marker becomes obscured or movesoutside of the field of view of all cameras, its six-degree-of-freedomposition is automatically reconstructed based on the visible markers.Thus, even if the primary marker happens to be obscured at that time,mapping other markers (moving rigidly and relative to each other) to thehistorical relative position of the primary marker allows for theprimary marker's location to be reconstructed.

Tracking System Calibration Overview

This disclosure previously described some embodiments utilizing anexample handling of the lens calibration, which involved the correctionof a barrel distortion in the lens based on prior lens calibrationmeasurements. However, a more sophisticated lens calibration method maybe used in combination with the generalized, six-degree-of-freedomextraction algorithm.

In some embodiments, there may be a separate lens calibration that isperformed with a checkerboard target (or in some embodiments anothertype of target). The checkerboard target may have a structure thatcomprises a set of nine marker structures arranged in a grid pattern (ora different number of marker structures). A user may reorient thecheckerboard target based on where the desired zero origin should be,then the cameras may image the checkerboard target. The calibrationalgorithm may then be used to optimize camera positions instead ofprincipal quantities.

In some embodiments, a radial component and a tangential component ofthe barrel distortion may be stored in the camera, which may be specificto a particular camera-lens combination. The system may guess a locationfor the target in the world coordinate system, and then project themarker, based what may be expected to be seen in the images. Thepreviously-calculated distortion may be applied, and then the projectionmay be converted into pixels and paired with the actual data that wascollected from the sensor.

Tracking System Calibration Setup

After each lens is assembled to a camera and a lens calibration isperformed to determine its focal length and distortion coefficients, thecameras are integrated with MR system and the integrated system iscalibrated as described herein. Although this example is described withrespect to cameras integrated into an MR system, similar calibrationconcepts may be utilized with other camera setups.

FIG. 39 shows the orientation and positions of components in anembodiment of a MR system, with first camera 3902, second camera 3904,and reference target 3906.

The integrated camera positions (defined as the center of the focalplanes) are taken to lie at (x_(cj),y_(cj),z_(cj)). The cameras' (3902and 3904) optical axes are pointed at the nominal target centerposition, (x₀,y₀,z₀), along the vectors(x₀−x_(cj))ê_(x)+(y₀−y_(cj))ê_(y)+(z₀−z_(cj))ê_(z). The focal planesthen lie in the laboratory planes(x₀−x_(cj))(x−x_(cj))+(y₀−y_(cj))(y−y_(cj))+(z₀−z_(cj))(z−z_(cj))=0.

Now, by convention for definition of angles, the cameras are oriented(spun as necessary about their optical axes) such that the vertical(“Y”) axis of each focal plane lays in the laboratory plane z=z_(cj);these vertical axes are then defined by the lines

(x ₀ −x _(cj))(x−x _(cj))+(y ₀ −y _(cj))(y−y _(cj))=0, or equivalently

x(x ₀ −x _(cj))+y(y ₀ −y _(cj))=x _(cj)(x ₀ −x _(cj))+y _(cj)(y ₀ −y_(cj))

The focal plane vertical “Y” axis unit vectors

$\begin{matrix}{{{\hat{Y}}_{cj} = \frac{{\left( {y_{0} - y_{c\; 1}} \right){\hat{e}}_{x}} - {\left( {x_{0} - x_{c\; 1}} \right){\hat{e}}_{y}}}{\sqrt{\left( {x_{0} - x_{c\; 1}} \right)^{2} + \left( {y_{0} - y_{c\; 1}} \right)^{2}}}}\left( {{for}\mspace{14mu} {cameras}\mspace{14mu} {to}\mspace{14mu} {the}\mspace{14mu} {{patient}'}s\mspace{14mu} {left}\mspace{14mu} {side}} \right)} & {1B} \\{{{\hat{Y}}_{cj} = \frac{{{- \left( {y_{0} - y_{c\; 2}} \right)}{\hat{e}}_{x}} + {\left( {x_{0} - x_{c\; 2}} \right){\hat{e}}_{y}}}{\sqrt{\left( {x_{0} - x_{c\; 2}} \right)^{2} + \left( {y_{0} - y_{c\; 2}} \right)^{2}}}}\left( {{for}\mspace{14mu} {cameras}\mspace{14mu} {to}\mspace{14mu} {the}\mspace{14mu} {{patient}'}s\mspace{14mu} {right}\mspace{14mu} {side}} \right)} & \;\end{matrix}$

are approximately antiparallel to the laboratory x axis, in the usualcase where x₀≈x_(cj). Taking the vector cross product of the opticalaxis unit vectors

$\frac{{\left( {x_{0} - x_{cj}} \right){\hat{e}}_{x}} + {\left( {y_{0} - y_{cj}} \right){\hat{e}}_{y}} + {\left( {z_{0} - z_{cj}} \right){\hat{e}}_{z}}}{\sqrt{\left( {x_{0} - x_{cj}} \right)^{2} + \left( {y_{0} - y_{cj}} \right)^{2} + \left( {z_{0} - z_{cj}} \right)^{2}}}$

with the camera vertical axis vectors gives the camera focal planehorizontal axis vectors:

$\begin{matrix}{{\hat{X}}_{cj} = {{- \frac{\begin{matrix}{{\left( {z_{0} - z_{c\; 1}} \right)\left( {x_{0} - x_{c\; 1}} \right){\hat{e}}_{x}} + {\left( {z_{0} - z_{c\; 1}} \right)\left( {y_{0} - y_{c\; 1}} \right){\hat{e}}_{y}} -} \\{\left\lfloor {\left( {x_{0} - x_{c\; 1}} \right)^{2} + \left( {y_{0} - y_{c\; 1}} \right)^{2}} \right\rfloor {\hat{e}}_{z}}\end{matrix}}{\begin{matrix}\sqrt{\left( {x_{0} - x_{c\; 1}} \right)^{2} + \left( {y_{0} - y_{c\; 1}} \right)^{2} + \left( {z_{0} - z_{c\; 1}} \right)^{2}} \\\sqrt{\left( {x_{0} - x_{c\; 1}} \right)^{2} + \left( {y_{0} - y_{c\; 1}} \right)^{2}}\end{matrix}}}\mspace{14mu} \left( {{left}\mspace{14mu} {side}} \right)}} & {2B} \\{{\hat{X}}_{cj} = {\frac{\begin{matrix}{{\left( {z_{0} - z_{c\; 2}} \right)\left( {x_{0} - x_{c\; 2}} \right){\hat{e}}_{x}} + {\left( {z_{0} - z_{c\; 2}} \right)\left( {y_{0} - y_{c\; 2}} \right){\hat{e}}_{y}} -} \\{\left\lfloor {\left( {x_{0} - x_{c\; 2}} \right)^{2} + \left( {y_{0} - y_{c\; 2}} \right)^{2}} \right\rfloor {\hat{e}}_{z}}\end{matrix}}{\begin{matrix}\sqrt{\left( {x_{2} - x_{c\; 2}} \right)^{2} + \left( {y_{0} - y_{c\; 2}} \right)^{2} + \left( {z_{0} - z_{c\; 2}} \right)^{2}} \\\sqrt{\left( {x_{0} - x_{c\; 2}} \right)^{2} + \left( {y_{0} - y_{c\; 2}} \right)^{2}}\end{matrix}}\mspace{14mu} \left( {{right}\mspace{14mu} {side}} \right)}} & \;\end{matrix}$

In the simple embodiment for which x_(cj)=x₀, the “X” and “Y” axis unitvectors reduce to

${{\hat{X}}_{cj} = \frac{{\left( {z_{0} - z_{cj}} \right){\hat{e}}_{y}} - {\left( {y_{0} - y_{cj}} \right){\hat{e}}_{z}}}{\sqrt{\left( {y_{0} - y_{cj}} \right)^{2} + \left( {z_{0} - z_{cj}} \right)^{2}}}};{{\hat{Y}}_{cj} = {{- 1}{\hat{e}}_{x}}}$

After the cameras are mounted in the system, a reference target isplaced approximately at the desired camera system origin and roughlyaligned to the desired camera coordinate system x and y axes.

FIG. 40 illustrates one embodiment of the reference target used intracking system calibration. The reference target pattern showncomprises 9 tracking marker patterns arranged in a 3×3 grid, spaced 25mm apart center to center. Additional markings indicate the properorientation and provide alignment marks for use with laser crosshairs orsimilar alignment tools. Various other reference target patterns may beutilized.

The camera coordinate system origin typically corresponds to a positionat or near the MR system isocenter. The calibration target positioningand alignment can be easily performed using laser cross-hairs availablein many MR systems. Camera parameters, aside from lens focal length anddistortion coefficients, which have been measured previously, includephysical location of focal plane centers and angular orientationsrelative to the reference target center:(x_(cj),y_(cj),z_(cj),α_(cj),γ_(cj),ω_(cj)). Here α_(cj) are the polarangles of the camera optical axes relative to nadir, γ_(cj) are theazimuth angles of these optical axes, and ω_(cj) define any further“twist” rotation of the cameras about their optical axes (relative tothe nominal focal plane orientation convention described in the boxedtext above).

FIG. 43 illustrates the nominal polar angle 4302, nominal azimuth angle4304, and the nominal twist angle 4306 of a camera.

FIG. 44 illustrates the focal plane of a camera for a case whenx_(c1)=x₀.

Nominal camera positions (x_(cj),y_(cj),z_(cj)) are chosen withconsideration for viewing access through various models of MRI headcoils. By convention, the nominal polar angle is taken to be

${\alpha_{cj} = {\tan^{- 1}\frac{\sqrt{\left( {\alpha_{cj} - \alpha_{0}} \right)^{2} + \left( {y_{cj} - y_{0}} \right)^{2}}}{z_{cj} - z_{0}}}},$

the nominal azimuth angle is taken to be

${\gamma_{cj} = {\tan^{- 1}\left( \frac{y_{0} - y_{cj}}{x_{0} - x_{cj}} \right)}},$

(using the usual convention for the arctangent function, for which theangle falls in the first quadrant, or 0 to 90 degrees, if both numeratorand denominator are positive, in the second quadrant if the numerator ispositive and denominator negative, and so on), and the nominal twistangle (relative to the reference orientation described in boxed textabove) is taken to be

ω_(cj)=0.

Tracking System Calibration Algorithm

In some embodiments, a calibration algorithm starts by determining thepoint on the z=z_(c) plane which projects to the center of the focalplane. This point is unaffected by camera twist, but depends on allother camera variables as follows:

$\begin{matrix}{{{x_{0,j} = {x_{cj} + \frac{\left( {z_{cj} - z_{0}} \right)\cos \; \gamma_{cj}}{\tan \; \alpha_{cj}}}};}{y_{0,j} = {y_{cj} + \frac{\left( {z_{cj} - z_{0}} \right)\sin \; \gamma_{cj}}{\tan \; \alpha_{cj}}}}} & {3B}\end{matrix}$

The point (x_(0,j), y_(0,j), z₀) becomes the apparent coordinate originfor camera j, but the laboratory coordinates (x₀,y₀,z₀) of the targetcenter in its home position on the calibration station are fixed, bydefinition, at this absolute point in space. At the home position, or atany other setting for the six translation/rotation degrees of freedom,the location of the target centroids can be calculated precisely inlaboratory coordinates, with the assumption of no positional error dueto the confirmed accuracy of the calibration station.

Defining several vectors which are used in determining the location onthe camera focal planes (X_(t), Y_(t)) of the image projection from anarbitrary reference target centroid point (x_(t),y_(t),z_(t)) inlaboratory coordinates:

The vector from the apparent origin of coordinates for either camera tothe camera itself is:

{right arrow over (R)} _(0,cj)=(x _(cj) −x _(0,cj))ê _(x)+(y _(cj) −y_(0,cj))ê _(y)+(z _(cj) −z ₀)ê _(z)

The vector between the apparent origin and the true target position is:

{right arrow over (R)} _(t,cj)=(x _(t) −x _(0,cj))ê _(x)+(y _(t) −y_(0,cj))ê _(y)+(z _(t) −z ₀)ê _(z)  4B

FIG. 45 illustrates a vector between the apparent origin and the pointof intersection of the camera optical axis with the perpendicular planethat contains the target point.

The vector between the apparent origin and the point of intersection ofthe camera optical axis with the perpendicular plane that contains thetarget point (see figure above) is:

${\overset{\rightharpoonup}{R}}_{\bot{,{cj}}} = {{\frac{{{\overset{\rightharpoonup}{R}}_{t,{cj}} \cdot {\overset{\rightharpoonup}{R}}_{0,{cj}}}}{{{\overset{\rightharpoonup}{R}}_{0,{cj}}}^{2}}{\overset{\rightharpoonup}{R}}_{0,{cj}}} = {\kappa \left\lbrack {{\left( {x_{cj} - x_{0,{cj}}} \right){\hat{e}}_{x}} + {\left( {y_{cj} - y_{0,{cj}}} \right){\hat{e}}_{y}} + {\left( {z_{cj} - z_{0}} \right){\hat{e}}_{z}}} \right\rbrack}}$     where$\kappa_{cj} = \frac{{\left( {x_{cj} - x_{0,{cj}}} \right)\left( {x_{t} - x_{0,{cj}}} \right)} + {\left( {y_{cj} - y_{0,{cj}}} \right)\left( {y_{t} - y_{0,{cj}}} \right)} + {\left( {z_{cj} - z_{0}} \right)\left( {z_{t} - z_{0}} \right)}}{\left( {x_{cj} - x_{0,{cj}}} \right)^{2} + \left( {y_{cj} - y_{0,{cj}}} \right)^{2} + \left( {z_{cj} - z_{0}} \right)^{2}}$     so  that     x_(⊥, cj) = x_(0, cj) + κ_(cj)(x_(cj) − x_(0, cj)); y_(⊥, cj) = y_(0, cj) + κ_(cj)(y_(cj) − y_(0, cj));     z_(⊥, cj) = z₀ + κ_(cj)(z_(cj) − z₀).

The unit vector running from the optical axis intercept to the targetcentroid in the perpendicular plane is

${\hat{r}}_{\bot{,{cj}}} = \frac{{\left( {x_{t} - x_{\bot{,{cj}}}} \right){\hat{e}}_{x}} + {\left( {y_{t} - y_{\bot{,{cj}}}} \right){\hat{e}}_{y}} + {\left( {z_{t} - z_{\bot{,{cj}}}} \right){\hat{e}}_{z}}}{\sqrt{\left( {x_{t} - x_{\bot{,{cj}}}} \right)^{2} + \left( {y_{t} - y_{\bot{,{cj}}}} \right)^{2} + \left( {z_{t} - z_{\bot{,{cj}}}} \right)^{2}}}$

Simple trigonometry from the figure above shows that the polar anglebetween the camera's optical axis and the true target point is:

$\theta_{t,{cj}} = {\sin^{- 1}\left( \frac{\sqrt{\left( {x_{t} - x_{\bot{,{cj}}}} \right)^{2} + \left( {y_{t} - y_{\bot{,{cj}}}} \right)^{2} + \left( {z_{t} - z_{\bot{,{cj}}}} \right)^{2}}}{\sqrt{\left( {x_{cj} - x_{t}} \right)^{2} + \left( {y_{cj} - y_{t}} \right)^{2} + \left( {z_{cj} - z_{t}} \right)^{2}}} \right)}$

The azimuth angle of the target point relative to the focal planevertical axis is determined by projection. In any plane perpendicular tothe optical axis, the focal plane principal axis ({circumflex over (X)}and Ŷ) unit vectors are given by equations (2) and (1) respectively; inthe absence of camera twist, the focal plane azimuth angle β isdetermined through the dot product of the r_(⊥) vector with theprojected vertical focal plane (Ŷ) vector. Adding the camera twist(minus sign by convention and definition only),

$\beta_{t,{cj}} = {{\pm {\cos^{- 1}\left( \frac{{\left( {x_{t} - x_{\bot{,{cj}}}} \right)\left( {y_{0,{cj}} - y_{cj}} \right)} - {\left( {y_{t} - y_{\bot{,{cj}}}} \right)\left( {x_{0,{cj}} - x_{cj}} \right)}}{\begin{matrix}\sqrt{\left( {x_{t} - x_{\bot{,{cj}}}} \right)^{2} + \left( {y_{t} - y_{\bot{,{cj}}}} \right)^{2} + \left( {z_{t} - z_{\bot{,{cj}}}} \right)^{2}} \\\sqrt{\left( {x_{0,{cj}} - x_{cj}} \right)^{2} + \left( {y_{0,{cj}} - y_{cj}} \right)^{2}}\end{matrix}} \right)}} - {\omega_{cj}\mspace{14mu} \left( {{left}\mspace{14mu} {side}} \right)}}$$\beta_{t,{cj}} = {{\pm {\cos^{- 1}\left( {- \frac{{\left( {x_{t} - x_{\bot{,{cj}}}} \right)\left( {y_{0,{cj}} - y_{cj}} \right)} - {\left( {y_{t} - y_{\bot{,{cj}}}} \right)\left( {x_{0,{cj}} - x_{cj}} \right)}}{\begin{matrix}\sqrt{\left( {x_{t} - x_{\bot{,{cj}}}} \right)^{2} + \left( {y_{t} - y_{\bot{,{cj}}}} \right)^{2} + \left( {z_{t} - z_{\bot{,{cj}}}} \right)^{2}} \\\sqrt{\left( {x_{0,{cj}} - x_{cj}} \right)^{2} + \left( {y_{0,{cj}} - y_{cj}} \right)^{2}}\end{matrix}}} \right)}} - {\omega_{cj}\mspace{14mu} \left( {{right}\mspace{14mu} {side}} \right)}}$

The arc-cos function is assumed to be bounded at 0 and 180 degrees, andthe appropriate sign multiplier for the azimuth angle is given by thesign of the projection (dot product) of the target perpendicular on thefocal plane horizontal axis vector, as

sign[β_(t,cj)]=sign(r _(⊥,cj) ·{circumflex over (X)});

sign[β_(t,cj)]=sign[(z _(0j) −z ₀){(x _(0,cj) −x _(cj))(x _(t) −x_(⊥,cj))+(y _(0,cj) −y _(cj))(y _(t) −y _(⊥,cj))}+(z _(t) −z _(⊥,cj)){(x_(0,cj) −x _(cj))²+(y _(0,cj) −y _(cj))²}]  (left side)

sign[β_(t,cj)]=−sign[(z _(0j) −z ₀){(x _(0,cj) −x _(cj))(x _(t) −x_(⊥,cj))+(y _(0,cj) −y _(cj))(y _(t) −y _(⊥,cj))}+(z _(t) −z _(⊥,cj)){(x_(0,cj) −x _(cj))²+(y _(0,cj) −y _(cj))²}]  (right side)

The camera projections of these new target centroid positions aredetermined using a geometric projection calculation. The normalizedpinhole camera projections are given by:

x _(t,cj)=

tan α_(t,cj) sin β_(t,cj) ,y _(t,cj)=±tan α_(t,cj) sin β_(t,cj)

Radial and tangential lens distortion effects are then applied to thenormalized projections according to the following equation:

$\begin{bmatrix}x_{t,{{cj}\mspace{14mu} {distorted}}} \\y_{t,{{cj}\mspace{14mu} {distorted}}}\end{bmatrix} = {{\left( {1 + {ar}^{2} + {br}^{4} + {cr}^{6}} \right)\begin{bmatrix}x_{t,{cj}} \\y_{t,{cj}}\end{bmatrix}} + \begin{bmatrix}{{2{dx}_{t,{cj}}y_{t,{cj}}} + {e\left( {r^{2} + {2x_{t,{cj}}^{2}}} \right)}} \\{{d\left( {r^{2} + {2y_{t,{cj}}^{2}}} \right)}2{ex}_{t,{cj}}y_{t,{cj}}}\end{bmatrix}}$

Where r²=x²+y², and the radial distortion coefficients, a, b, c and thetangential distortion coefficients d, e, are determined by a separatelens calibration procedure. Lastly, the distorted projection coordinatesare mapped to pixel values for each camera as follows:

${X_{t,{cj}} = {X_{0} + {\left( \frac{f.l.}{s_{pix}} \right)x_{t,{{cj}\mspace{14mu} {distorted}}}}}},{Y_{i,j} = {Y_{0} + {\left( \frac{f.l.}{s_{pix}} \right)y_{t,{{cj}\mspace{14mu} {distorted}}}}}}$

Where f.l. and are s_(pix) the lens focal length and camera pixel pitch,and X₀ and Y₀ are the pixel coordinates of the center of the pixel array(i.e. for a 1280×1024 array, X₀=640.5 and Y₀=512.5, assuming the centerof the upper left corner pixel is defined as having coordinates (1,1).).

Here X_(0,cj) and Y_(0,cj) are the horizontal and vertical number of thecentral pixel column and row of camera j (assumed to intersect theoptical axis of that camera), and f.l._(cj) and s_(pix) are the camera jlens focal length and pixel pitch.

Tracking System Calibration Routine

After assembly and rough system alignment, a calibration routine is runto determine residual errors in camera orientations(ôx_(cj),ôy_(cj),ôz_(cj),ôα_(cj),ôγ_(cj),ôω_(cj)).

A small error displacement ôx_(cj) of a camera in the x-direction or asmall error rotation ôγ_(cj) of the camera optical axis about thelaboratory “z” axis will result in the center of the reference target(i.e. home position) being imaged at a displaced position relative tothe vertical center of the focal plane. Likewise, small errordisplacements ôy_(cj) and/or ôz_(cj) of camera position in the y- and/orz-directions will result in the center of the reference target beingimaged at a position displaced relative to the horizontal center of thefocal plane. By viewing multiple target features simultaneously, (e.g.the individual centroids of the reference target), image displacementscorresponding to errors in camera position or orientation along sixdegrees of freedom can be distinguished using a similar method to thatused for in the Head Tracker 6DOF extraction algorithm.

The self-calibration algorithm considers the effects of small errors ineach of the camera orientation parameters on the projected centroidpositions seen by each camera. The reference target center defines theorigin of the camera coordinate system and its orientation defines thecoordinate system axes. This becomes the basis of measurement for cameraposition/orientation errors. The reference target itself is assumed toprovide truth data relative to size scale. Thus, by definition, the truepositions of the reference target centroids are known in laboratorycoordinates, and by extension, the expected locations on the camerafocal planes for projections of these points are also known.

Comparison of the measured pixel positions (X_(t,cj),tY_(t,cj)) of theeach of the imaged centroids on each camera, with the expectedprojections gives 2N equations for the six unknown orientation errors(ôx_(cj),ôy_(cj),ôz_(cj),ôα_(cj),ôγ_(cj),ôω_(cj)) for each camera, whereN is the number of unique centroids in the reference target pattern.Like the run-time head tracker software, the calibration algorithm isbased on computing the local derivatives of the (X_(t,cj),Y_(t,cj))centroid projections with respect to each error axis, and thenperforming a matrix inversion to determine the errors themselves. Thecalculation is performed iteratively until convergence to within anacceptable tolerance is achieved.

The procedure outlined above only requires the end user to place andorient the reference target once and then start the calibration routine,after which all calculations are performed without additional userinteraction. The resulting coordinates for all cameras are then storedin the system. Camera position calibrations need only be performedperiodically or when changes to camera positions are suspected.

Additional Features

Adapting to Addition/Removal of Cameras (e.g., Camera Viewpaths beingUnblocked/Blocked)

In some embodiments there may be a plurality of cameras. However, incertain situations one or more cameras in the plurality of cameras maystop working or have their viewpath obscured, or otherwise be incapableof detecting an image of a marker. It may be helpful for the system toutilize a generalized six-degree-of-freedom algorithm, as disclosedherein, that allows the system to continue performing when a camera goesdown (e.g., by having its viewpath obscured). The same method or processmay also enable the seamless addition of cameras (e.g., when a viewpathbecomes unobstructed) into the motion compensation system. Thus,desirably the motion tracking is not disturbed when a camera goes down,and it is also not disturbed when the camera comes back online.

FIG. 41 is a flow chart that illustrates an example embodiment of howthe system may adapt to the addition or removal of cameras/markers (e.g.a view path becoming unobstructed or obstructed) using the generalizedsix-degrees-of-freedom algorithm. As used herein, the terms adding orremoving a camera may refer to a camera's view path to a marker becomingunobstructed or obstructed, respectively. These terms may also refer toa camera being activated or deactivated, physically added or removedfrom the system, and/or the like. At block 4102, an additional cameramay be added to the system. Alternatively, at block 4104 a camera may beremoved from the system. The removal may be intentional, such as if acamera is malfunctioning, or the removal may be a result of an accidentor patient motion, such as if the patient raises his or her arm in frontof one of the cameras, rotates his or her head, and/or the like,resulting in a blocked viewpath of that camera. Regardless of whathappened, for each marker the system at block 4106 determines whichremaining cameras can see that marker. At block 4108, the system thenuses image data only from cameras with an unobstructed viewpath with agiven marker, which was determined at block 4106.

Then, at block 4109 the system generates a set of principal quantities.The system calculates the set of principal quantities by calculatingboth the sum and difference of the X (column) and Y (row) coordinatesbetween each target image and every other target image across allcameras. Thus, a set of principal quantities may be constructed for eachmarker based on all available image data.

The system at block 4110 numerically evaluates the partial derivativesrelative to subject displacements and rotations of the principalquantities. More specifically, local partial derivatives of each of theprincipal quantities, with respect to each of the six degrees offreedom, are determined numerically by evaluating changes in theprincipal quantities for small increments in each degree of freedom. Thesystem at block 4112 may perform a calibration procedure, and thatcalibration may be used to help numerically evaluate the partialderivatives of the principal quantities about the initialized targetposition. In some embodiments, the calibration procedure is onlyperformed once, or only periodically. The calibration procedure does nothave to be performed every time a patient is tracked. The system obtainsa partial derivative matrix at the end of block 4110.

At block 4114, the system may then determine a first-order approximationof the displacement matrix by multiplying the matrix of measuredprincipal quantities, obtained following block 4109, with the inverse ofthe partial derivative matrix obtained at the end of block 4110.

At block 4116, the system may then generate a residual error matrix. Theresidual error matrix may be obtained from taking the first-orderapproximations, entering them into a translation equation to determinethe corresponding translated 3-D target position for each of the targetpositions, obtaining 3-D world coordinates that are projected to cameracoordinates, and re-calculating the principal quantities. Theseprincipal quantities may be compared against the measured values of theprincipal quantities which were obtained after block 4109, and thedifferences would reside in the residual error matrix. A local partialderivative matrix can then be obtained, with respect to each of the sixdegrees of freedom, by calculating the effect of small increments ineach of the six degrees of freedom relative to the first-orderdisplacement vector. For each increment, the new world coordinates andthe new camera projections of the target positions are re-computed andthe principal quantities are re-calculated. The change in each principalquantity is divided by the small angular or displacement increment todetermine a local partial derivative. Repeating this for each of the sixdegrees of freedom yields the six-column local partial derivativematrix, as each repetition produces one column.

At block 4118, the system may then compute a coarse correction toimprove the first-order displacement vector and reduce residual error,by multiplying the residual error matrix from block 4116 with theinverse of the local partial derivative matrix from block 4116. Thesystem may then increment the first-order displacement vector by thecoarse correction matrix to create a better approximation of thedisplacement vector.

At block 4120, the system may then perform an interative finecorrection. The system continue looping back to block 4116 to determinea fine correction to the displacement vector. With each iteration, theresultant fine correction increments are added to the coarse-correctedvector to create the new displacement vector. The steps are repeatediteratively until the magnitude of each of the fine corrections is lessthan some threshold, or a number of pre-defined iterations have beenperformed. Afterwards, at block 4122 the system arrives at a finaldisplacement vector, which may be associated with the position of themarker.

Additional Process Flow

If a patient is sitting still, or moving very slowly, noise in thetracking data may cause uncertainty or noise in the image correction,making it possible for the image correction to do more harm than good.It may be insufficient to simply turn off the image correction and keepthe patient centered on the origin, since a patient may turn his headduring the procedure and end up falling asleep in a position with alarge offset. Thus, it may be helpful to perform an image correctionthat may be able to filter out any noise if there is very little motionin the patient.

In some embodiments, prior to reporting the Six-Degree-of-Freedom markerposition data (e.g., prior to sending tracking data to a controller ofthe scanner for compensation for motion), a noise filter or outputfilter may be applied to smooth the fluctuations in coordinate valueswhen there is little to no active motion.

In some embodiments, this filter comprises an application of an adaptiveKalman filter. The Kalman filter may use a constant velocity kinematicmodel, in which the process noise (that represents the uncertainty ofthe model) is in the form of an unknown acceleration. The magnitude ofthe process noise may control the relative weighting of the model vs.the measured data in the filter output.

In some embodiments, the new, measured data may be compared to theaverage of the last 10 data points and a scale factor dependent on thisdifference may be applied to the process noise input of the Kalmanfilter. This may cause the model to be strongly preferred when the newdata falls within 3σ of the recent average (of the last 10 data points),and continuously shifts preference toward the data as the differenceincreases beyond this range. Thus, if there is a large process noise,the filter will strongly weight the new data coming in and will hardlydo any filtering at all of this data. If there is very little processnoise, the filter may prefer the model over the data and perform a verystrong smoothing or averaging of the data. Thus, the resulting filterstrongly smooths the data when there is little to no motion by thepatient, yet responds instantly to large changes in position withoutovershoot or lag. If the patient performs a large, instantaneous motion,the filter adapts right away and will go with the new data coming inthat is associated with that large motion.

FIG. 42 is a block diagram depicting another embodiment of a motioncompensation system 4201. The embodiment in FIG. 42 is similar to theembodiment of the motion compensation system 300 illustrated in FIG. 3B.

Motion compensation system 4201 also utilizes one or more detectors 108.However, motion compensation 4201 is configured to use asix-degree-of-freedom algorithm that has been generalized so that motioncompensation system 4201 may use any number, from 1 to n, of detectors108. These detectors 108 may be configured to be used with any number ofmarkers. The figure shows marker 1 (110-1), marker 2 (110-2), all theway up to marker n (110-n). These markers may have slight variations inthem that allow the markers to be uniquely identifiable from oneanother. Each marker may at different times be imaged by zero, one, two,more than two, or all the detectors 108.

Calibration engine 4206 may comprise a calibration database 4216, adetector calibration filter 4218, and a target calibration filter 4220.In some embodiments, calibration engine 4206 may be configured tocalibrate the system at the initial startup of the system. In otherembodiments, calibration engine 4206 may be used for at least somecalibration procedures during some or all motion tracking procedures.The detector calibration filter 4218 may be configured to calibrate thedetectors 108 to the motion compensation system. In some embodiments,detector calibration filter 4218 may be configured to allow amanufacturer or assembler of a motion compensation system to inputinformation specific to each detector 108, such as values associatedwith barrel distortion for the detector 108. In some embodiments, targetcalibration filter 4220 may be be configured to calibrate the system toone or more specific targets or markers, such as a grid of nine markershaving the embodiment shown in FIG. 34. This information may be storedin calibration database 4216, and may help speed up a tracking routine.

Tracking engine 4234 may comprise a reference point filter 4236, aprincipal quantities generator 4238, a reference frame translator 4240,an error filter 4242, a convergence filter 4244, and an output filter4246. The components of the tracking engine 4234 may be configured tooperate similarly to the process flow described in association with FIG.7C and FIG. 41. However, there may be some slight variation.

For example, the reference point filter 4236 may be configured toanalyze determine the locations of, and identify, the reference pointsin each marker. This information may be used to determine if anydetectors 108 are not detecting or imaging a certain marker, and thenexclude the image data from that detector from calculation. Principalquantitates generator 4238 may be configured to determine a set ofprincipal quantities, which are obtained by calculating both the sum anddifference of the X (column) and Y (row) coordinates between each targetimage and every other target image across all cameras. The principalquantities may be constructed for each marker based on all availableimage data.

The reference frame translator 340 may be configured to convert betweenthe two dimensional reference frame of each detector 108 and the threedimensional frame of the motion tracking system 4203. The error filter4242 may be configured to analyze differences in principal quantitiesbased on the visualized reference points and based on estimates of anobject's pose to determine an amount of error between the two. Theconvergence filter 4244 may be configured to perform an iterativeprocess to reduce an amount of error in an object pose estimate until anobject pose estimate has an acceptable amount of error.

Output filter 4246 may be the adaptive Kalman filter that was previouslydiscussed. Output filter 4246 may be configured to detect when a patientis not moving or exhibiting little movement, and then perform a strongsmoothing or averaging of the data in order to reduce the impact ofnoise. Output filter 4246 may also be configured to detect when apatient is making large movements, and respond the large movementswithout overshoot or lag. At block 4222, the coordinate system convertermay remap the coordinate system to the same axes as the MR coordinatesystem, and convert to a quaternion representation of the angles.

Variations

Specific embodiments have been described in detail above with emphasison medical application and in particular MRI examination of a patient'shead. However, the teachings of the present invention can be utilizedfor other MRI examinations of other body parts where movements of up tosix degrees of freedom are possible. In addition medical proceduresinvolving imaging devices other than MRI equipment (e.g., CT, PET,ultrasound, plain radiography, and others) may benefit from the teachingof the present invention. The teachings of the present invention may beuseful in many non-medical applications where tracking of a targethaving several degrees of freedom are possible. Some of theseapplications could be military applications. Furthermore, whileparticular algorithms are disclosed, variations, combinations, andsubcombinations are also possible.

Computing System

In some embodiments, the computer clients and/or servers described abovetake the form of a computing system 1500 illustrated in FIG. 21, whichis a block diagram of one embodiment of a computing system that is incommunication with one or more computing systems 1520 and/or one or moredata sources 1522 via one or more networks 1518. The computing system1500 may be used to implement one or more of the systems and methodsdescribed herein. In addition, in one embodiment, the computing system1500 may be configured to apply one or more of the methods and systemsdescribed herein. While FIG. 21 illustrates an embodiment of a computingsystem 1500, it is recognized that the functionality provided for in thecomponents and modules of computing system 1500 may be combined intofewer components and modules or further separated into additionalcomponents and modules.

Motion Correction Control Systems

In an embodiment, the system 700 comprises a motion correction controlsystem module 1514 that carries out the functions described herein withreference to motion correction mechanism, including any one of themotion correction methods described above. The motion correction controlsystem module 1514 may be executed on the computing system 1500 by acentral processing unit 1504 discussed further below.

In general, the word “module,” as used herein, refers to logic embodiedin hardware or firmware, or to a collection of software instructions,possibly having entry and exit points, written in a programminglanguage, such as, for example, COBOL, CICS, Java, Lua, C or C++ orObjective C. A software module may be compiled and linked into anexecutable program, installed in a dynamic link library, or may bewritten in an interpreted programming language such as, for example,BASIC, Perl, or Python. It will be appreciated that software modules maybe callable from other modules or from themselves, and/or may be invokedin response to detected events or interrupts. Software instructions maybe embedded in firmware, such as an EPROM. It will be furtherappreciated that hardware modules may be comprised of connected logicunits, such as gates and flip-flops, and/or may be comprised ofprogrammable units, such as programmable gate arrays or processors. Themodules described herein are preferably implemented as software modules,but may be represented in hardware or firmware. Generally, the modulesdescribed herein refer to logical modules that may be combined withother modules or divided into sub-modules despite their physicalorganization or storage.

Computing System Components

In an embodiment, the computing system 1500 also comprises a workstationor other computing devices suitable for controlling and/or communicatingwith large databases, performing transaction processing, and generatingreports from large databases. The computing system 1500 also comprises acentral processing unit (“CPU”) 1504, which may comprise a conventionalmicroprocessor. The computing system 1500 further comprises a memory1508, such as random access memory (“RAM”) for temporary storage ofinformation and/or a read only memory (“ROM”) for permanent storage ofinformation, and a mass storage device 1502, such as a hard drive,diskette, or optical media storage device. Typically, the modules of thecomputing system 1500 are connected to the computer using a standardsbased bus system. In different embodiments, the standards based bussystem could be Peripheral Component Interconnect (PCI), Microchannel,SCSI, Industrial Standard Architecture (ISA) and Extended ISA (EISA)architectures, for example.

The computing system 1500 comprises one or more commonly availableinput/output (I/O) devices and interfaces 1512, such as a keyboard,mouse, touchpad, and printer. In one embodiment, the I/O devices andinterfaces 1512 comprise one or more display devices, such as a monitor,that allows the visual presentation of data to a user. Moreparticularly, a display device provides for the presentation of GUIs,application software data, and multimedia presentations, for example. Inthe embodiment of FIG. 21, the I/O devices and interfaces 1512 alsoprovide a communications interface to various external devices. Thecomputing system 1500 may also comprise one or more multimedia devices1506, such as speakers, video cards, graphics accelerators, andmicrophones, for example.

Computing System Device/Operating System

The computing system 1500 may run on a variety of computing devices,such as, for example, a mobile device or a server or a desktop or aworkstation, a Windows server, an Structure Query Language server, aUnix server, a personal computer, a mainframe computer, a laptopcomputer, a cell phone, a personal digital assistant, a kiosk, an audioplayer, a smartphone, a tablet computing device, and so forth. Thecomputing system 1500 is generally controlled and coordinated byoperating system software, such as iOS, z/OS, Windows 95, Windows 98,Windows NT, Windows 2000, Windows XP, Windows Vista, Windows 7, Linux,BSD, SunOS, Solaris, or other compatible operating systems. In Macintoshsystems, the operating system may be any available operating system,such as MAC OS X. In other embodiments, the computing system 1500 may becontrolled by a proprietary operating system. Conventional operatingsystems control and schedule computer processes for execution, performmemory management, provide file system, networking, and I/O services,and provide a user interface, such as a graphical user interface(“GUI”), among other things.

Network

In the embodiment of FIG. 21, the computing system 1500 is coupled to anetwork 1518, such as a LAN, WAN, or the Internet, for example, via awired, wireless, or combination of wired and wireless, communicationlink 1516. The network 1518 communicates with various computing devicesand/or other electronic devices via wired or wireless communicationlinks. In the embodiment of FIG. 21, the network 1518 is communicatingwith one or more computing systems 1520 and/or one or more data sources1522.

Access to the motion correction control system module 1514 of thecomputer system 1500 by computing systems 1520 and/or by data sources1522 may be through a web-enabled user access point such as thecomputing systems' 1520 or data source's 1522 personal computer,cellular phone, laptop, or other device capable of connecting to thenetwork 1518. Such a device may have a browser module is implemented asa module that uses text, graphics, audio, video, and other media topresent data and to allow interaction with data via the network 1518.

The browser module may be implemented as a combination of an all pointsaddressable display such as a cathode-ray tube (CRT), a liquid crystaldisplay (LCD), a plasma display, touch screen display or other typesand/or combinations of displays. In addition, the browser module may beimplemented to communicate with input devices 1512 and may also comprisesoftware with the appropriate interfaces which allow a user to accessdata through the use of stylized screen elements such as, for example,menus, windows, dialog boxes, toolbars, and controls (for example, radiobuttons, check boxes, sliding scales, and so forth). Furthermore, thebrowser module may communicate with a set of input and output devices toreceive signals from the user.

The input device(s) may comprise a keyboard, roller ball, pen andstylus, mouse, trackball, voice recognition system, or pre-designatedswitches or buttons. The output device(s) may comprise a speaker, adisplay screen, a printer, or a voice synthesizer. In addition a touchscreen may act as a hybrid input/output device. In another embodiment, auser may interact with the system more directly such as through a systemterminal connected to the score generator without communications overthe Internet, a WAN, or LAN, or similar network.

In some embodiments, the system 1500 may comprise a physical or logicalconnection established between a remote microprocessor and a mainframehost computer for the express purpose of uploading, downloading, orviewing interactive data and databases online in real time. The remotemicroprocessor may be operated by an entity operating the computersystem 1500, including the client server systems or the main serversystem, an/or may be operated by one or more of the data sources 1522and/or one or more of the computing systems. In some embodiments,terminal emulation software may be used on the microprocessor forparticipating in the micro-mainframe link.

In some embodiments, computing systems 1520 that are internal to anentity operating the computer system 1500 may access the motioncorrection control system module 1514 internally as an application orprocess run by the CPU 1504.

User Access Point

In an embodiment, the computing system 1500 comprises a computingsystem, a smartphone, a tablet computing device, a mobile device, apersonal computer, a laptop computer, a portable computing device, aserver, a computer workstation, a local area network of individualcomputers, an interactive kiosk, a personal digital assistant, aninteractive wireless communications device, a handheld computer, anembedded computing device, or the like.

Other Systems

In addition to the systems that are illustrated in FIG. 21, the network1518 may communicate with other data sources or other computing devices.The computing system 1500 may also comprise one or more internal and/orexternal data sources. In some embodiments, one or more of the datarepositories and the data sources may be implemented using a relationaldatabase, such as DB2, Sybase, Oracle, CodeBase and Microsoft® SQLServer as well as other types of databases such as, for example, asignal database, object-oriented database, and/or a record-baseddatabase.

FIG. 23 is an embodiment of a schematic diagram illustrating atherapeutic application system 1718 that is connected to one or moretracking systems. In an embodiment, a patient 1706 is receiving atherapeutic therapy to treat a condition. For example, the therapeutictherapy can be radiation therapy, proton therapy, or other therapies.The therapeutic therapy can be generated from a therapeutic applicationsystem 1718. For example, the therapeutic applications systems 1718 cancomprise a radiotherapy technology configured to generate a radiationbeam that is transmitted to a diseased tissue 1704. In an embodiment,the system comprises fiducial markers or other markers 1702. In anembodiment, the fiducial markers or other markers 1702 are placed insidethe patient 1706. The fiducial markers or other markers 1702 areconfigured to be detected by an electronics package 1710. The fiducialmarkers or other markers 1702 can be configured to identify the locationof the diseased tissue to be treated with the therapeutic therapy.

The electronics package 1710 is configured to detect the location of thefiducial marker or other marker 1702 in order to determine the locationof the diseased tissue 1704. In an embodiment, the electronics package1710 is coupled to a first marker tracking system 1708. The first markertracking system 1708 can be configured to receive tracking data from theelectronics package 1710 to determine the location of the markers 1702.By determining the locations of the markers 1702, the first markertracking system 1708 can be configured to determine the location of thediseased tissue 1704. In an embodiment, the systems and methodsdisclosed herein for tracking markers with 0.1 mm and 0.1 degreeaccuracies can be implemented or employed by the first marker trackingsystem 1708. In an embodiment, the electronics package 1710 comprises anoptical marker 1712. The optical marker 1712 is configured to bedetected by an optical scanner 1714, for example, a CCD camera. In anembodiment, the optical scanner 1714 is coupled to an optical markertracking system 1716.

The optical marker tracking system 1716 can be configured to determinethe location of the electronics package 1710 relative to the therapeuticapplication system 1718. In an embodiment, the systems and methodsdisclosed herein for tracking markers with 0.1 mm and 0.1 degreeaccuracies can be implemented or employed by the optical marker trackingsystem 1716. In an embodiment, the system can comprise a coordinategenerating system 1720 that is configured to receive tracking data fromthe first marker tracking system 1708 and the optical marker trackingsystem 1716. The coordinate generating system 1720 can be configured toanalyze the tracking data in order to generate coordinate data that canbe used to identify the location of the diseased tissue 1704. In anembodiment, the coordinate generating system 1720 can be configured totransmit the coordinate data to the therapeutic application system 1718.

The therapeutic application system 1718 can be configured to generate atherapeutic beam based on the coordinate data. For example, thetherapeutic application system 1718 can be configured to direct aradiation beam to a particular location in the patient 1706 based on thecoordinate data. Further, the therapeutic application system 1718 canalso be configured, for example, to generate a particular radiation beamshape based on the coordinate data. Any patient movement can be detectedby the electronics package 1710 and the optical scanner 1714. The firstmarker tracking system 1708 and the optical marker tracking system 1716can be configured to generate new tracking data to be inputted into thecoordinate generating system 1720. The coordinate generating system 1720can be configured to generate new coordinate data for transmission intothe therapeutic application system 1718. The therapeutic applicationsystem 1718 can be configured to analyze the new coordinate data inorder to redirect and/or reshape the therapeutic beam to be applied tothe diseased tissue 1704 of the patient 1706.

Similar to FIG. 23, in an embodiment, the system can comprise one motiontracking system as illustrated in FIG. 25. The single motion trackingsystem 1716, configured to employ the techniques disclosed herein, cantrack marker 1902 to generate coordinate data for transmission to atherapeutic application system 1718 in order to direct a therapeutictherapy to the target tissue site 1904.

FIG. 24 is a flowchart depicting an embodiment of a process for applyinga therapeutic therapy based on tracking patient movement. The processcan begin at block 1802 by optionally calibrating tracking systems withthe therapeutic application system at block 1804. At block 1806, thesystem can be configured to initiate the tracking systems. For example,the system can be configured to initiate the electronics packagetracking system and the optical marker tracking system. At block 1808,the system can be configured to track the first marker using theelectronics package. For example, the electronics package can beconfigured to track the fiducial marker that is implanted in thepatient. In an embodiment, at block 1810, the system can be configuredto determine the position of the marker relative to the target tissuesite.

At block 1818, the system can be configured to track the optical markerposition in order to determine the location of the electronics package.At block 1816, the system can be configured to determine the position ofthe electronics package relative to the therapeutic equipment. At block1818, the system can be configured to analyze the tracking data of thefirst marker and the optical marker, and generate coordinates of thetarget tissue site. The coordinates of the target tissue site can betransmitted to the therapeutic therapy equipment at block 1820. Thetherapeutic therapy equipment can be configured to utilize thecoordinate data to transmit the therapeutic therapy to the surgical siteat block 1820. At decision block 1822, the system can be configured torepeat the application of the therapeutic therapy. If the therapeutictherapy application should be repeated, the system can be configured toloop back to block 1806 to initiate the tracking systems. If thetherapeutic therapy application should not be repeated, the system canbe configured to end the process at block 1824.

Similar to FIG. 24, in an embodiment, the system can be configured toemploy an alternative method as illustrated in FIG. 26 wherein a singlecamera is used to track a marker in order to generate coordinate datafor use by the therapeutic application system.

Conditional language, such as, among others, “can,” “could,” “might,” or“may,” unless specifically stated otherwise, or otherwise understoodwithin the context as used, is generally intended to convey that certainembodiments include, while other embodiments do not include, certainfeatures, elements and/or steps. Thus, such conditional language is notgenerally intended to imply that features, elements and/or steps are inany way required for one or more embodiments or that one or moreembodiments necessarily include logic for deciding, with or without userinput or prompting, whether these features, elements and/or steps areincluded or are to be performed in any particular embodiment. Theheadings used herein are for the convenience of the reader only and arenot meant to limit the scope of the inventions or claims.

Although this invention has been disclosed in the context of certainpreferred embodiments and examples, it will be understood by thoseskilled in the art that the present invention extends beyond thespecifically disclosed embodiments to other alternative embodimentsand/or uses of the invention and obvious modifications and equivalentsthereof. Additionally, the skilled artisan will recognize that any ofthe above-described methods can be carried out using any appropriateapparatus. Further, the disclosure herein of any particular feature,aspect, method, property, characteristic, quality, attribute, element,or the like in connection with an embodiment can be used in all otherembodiments set forth herein. For all of the embodiments describedherein the steps of the methods need not be performed sequentially.Thus, it is intended that the scope of the present invention hereindisclosed should not be limited by the particular disclosed embodimentsdescribed above.

1. A motion tracking system for tracking and compensating for motion ofa patient during a medical imaging scan, the motion tracking systemcomprising: an optical marker configured to be attached to the patient,wherein the optical marker comprises a plurality of optically visiblelandmarks, the plurality of optically visible landmarks defining aplurality of reference points; a plurality of optical detectors, whereineach of the plurality of optical detectors is positioned to view theoptical marker along a different line of sight; a marker location filterfor determining locations of marker reference points, the markerlocation filter configured to: receive a digital image from each of theplurality of optical detectors, wherein each digital image represents aview along a different line of sight; determine, for each digital image,whether the digital image includes a view of the optical marker; and foreach digital image that includes a view of the optical marker, determinepositions of the plurality of reference points in the digital image; apatient orientation filter for estimating a three-dimensional pose ofthe patient, the patient orientation filter configured to: calculate aplurality of baseline attributes related to the plurality of referencepoints, the plurality of baseline attributes calculated based on thedetermined positions of the plurality of reference points; and estimateiteratively the three-dimensional pose of the patient, until a measureof error is within a threshold amount, the measure of error calculatedbased on the plurality of baseline attributes as compared to a pluralityof comparison attributes, the plurality of comparison attributescalculated by assuming the object is in an estimated pose; and one ormore computers configured to operate the marker location filter andpatient orientation filter, wherein the one or more computers comprisesa computer processor and an electronic storage medium.
 2. The motiontracking system of claim 1, wherein the patient orientation filter isconfigured to repeatedly estimate the three-dimensional pose of thepatient at a rate of at least 100 Hz.
 3. The motion tracking system ofclaim 1, wherein the plurality of baseline attributes comprises a set ofprincipal quantities, the set of principal quantities comprising valuesassociated with the relationships between the positions of referencepoints.
 4. The motion tracking system of claim 1, wherein determiningwhether each digital image includes the optical marker comprises:classifying each pixel in the digital image as one of two colors;grouping one or more connected regions, wherein the connected regionsare comprised of adjacent pixels with the some color; filtering out oneor more connected regions based on a size threshold; computing thecentroid of one or more connected regions; and grouping together one ormore connected regions by their computed centroids.
 5. The motiontracking system of claim 1, wherein the motion tracking system furthercomprises a coordinate system converter, the coordinate system converterconfigured to remap a coordinate system associated with thethree-dimensional pose of the patient estimated by the patientorientation filter.
 6. The motion tracking system of claim 4, whereingrouping together one or more connected regions comprises: identifyingconnected regions with centroids occurring within one pixel of eachother; and calculating a centroid for each group of the one or moreconnected regions.
 7. The motion tracking system of claim 4, wherein themotion tracking system binarizes each digital image before classifyingeach pixel in the digital image as white or black.
 8. The motiontracking system of claim 6, wherein calculating a centroid for eachgroup of the one or more connected regions comprises averaging thecentroid for each connected region in the group of one or more connectedregions.
 9. The motion tracking system of claim 1, further comprising: atracking filter configured to generate tracking data based on theestimated three-dimensional pose of the patient and to transmit thetracking data to a medical imaging scanner controller to enable amedical imaging scanner to dynamically adjust the medical imaging scanto compensate for patient motion.
 10. The motion tracking system ofclaim 9, further comprising: the medical imaging scanner, wherein themedical imaging scanner is configured to dynamically adjust the medicalimaging scan to compensate for patient motion based on the trackingdata.
 11. A motion tracking system for dynamic tracking of andcompensation for motion of a patient during a magnetic resonance scan,the motion tracking system comprising: an optical marker configured tobe attached to a patient being scanned; a first camera positioned toview the optical marker along a first line of sight; a second camerapositioned to view the optical marker along a second line of sight; anda computer system configured to analyze images generated by the firstand second cameras to determine changes in position of the opticalmarker, and to generate tracking data for use by a magnetic resonancescanner to dynamically adjust scans to compensate for the changes inposition of the optical marker, wherein the computer system isconfigured to dynamically determine whether the first camera, the secondcamera, or both cameras are currently viewing the optical marker, andwherein the computer system is configured to dynamically adapt its imageanalysis to utilize images from all cameras that are currently viewingthe optical marker.
 12. The motion tracking system of claim 11, furthercomprising: one or more additional cameras each positioned to view theoptical marker along a different line of sight, wherein the computersystem is further configured to dynamically determine which of all ofthe cameras are currently viewing the optical marker.
 13. The motiontracking system of claim 11, further comprising: a third camerapositioned to view the optical marker along a third line of sight; and afourth camera positioned to view the optical marker along a fourth lineof sight, wherein the computer system is further configured todynamically determine which of all of the cameras are currently viewingthe optical marker.
 14. The motion tracking system of claim 11, furthercomprising: one or more additional optical markers configured to beattached to the patient being scanned, wherein the computer system isconfigured to analyze images generated by all cameras that are currentlyviewing at least one optical marker to determine changes in position ofthe optical markers, and to generate tracking data for use by themagnetic resonance scanner to dynamically adjust scans to compensate forthe changes in position of the optical markers.
 15. The motion trackingsystem of claim 11, wherein the optical marker comprises a plurality ofoptically visible landmarks, the plurality of optically visiblelandmarks defining a plurality of reference points.
 16. The motiontracking system of claim 11, further comprising: the magnetic resonancescanner, wherein the magnetic resonance scanner is configured todynamically adjust scans to compensate for the changes in position ofthe optical marker based on the tracking data.
 17. The motion trackingsystem of claim 11, wherein the optical marker comprises: an opticallyvisible pattern comprising a plurality of reference point locators, eachreference point locator configured to define a single reference point,each reference point locator comprising alternating dark and lightelliptical shapes centered on the single reference point, wherein atleast one of the plurality of reference point locators is larger than atleast another of the plurality of reference point locators, wherein theoptically visible pattern is rotationally asymmetrical about an axisnormal to a plane of the optically visible pattern.
 18. (canceled) 19.(canceled)
 20. (canceled)
 21. (canceled)
 22. (canceled)
 23. (canceled)24. (canceled)
 25. (canceled)
 26. (canceled)
 27. A motion trackingsystem for dynamic tracking of and compensation for motion of a patientduring a magnetic resonance scan, the motion tracking system comprising:one or more optical marker configured to be attached to a patient beingscanned; one or more cameras positioned to comprise different lines ofsight; and a computer system configured to analyze images generated bythe one or more cameras to determine changes in position of the one ormore optical markers, and to generate tracking data for use by amagnetic resonance scanner to dynamically adjust scans to compensate forchanges in position of a patient based on the determined changes inposition of the one or more optical markers, wherein the computer systemis configured to dynamically determine which of the one or more camerasis currently viewing an optical marker, and wherein the computer systemis configured to dynamically adapt its image analysis to utilize imagesfrom all cameras that are currently viewing an optical marker.
 28. Themotion tracking system of claim 27, wherein each of the one or moreoptical markers comprises a plurality of optically visible landmarks,the plurality of optically visible landmarks defining a plurality ofreference points.
 29. The motion tracking system of claim 27, furthercomprising: the magnetic resonance scanner, wherein the magneticresonance scanner is configured to dynamically adjust scans tocompensate for the changes in position of the patient based on thetracking data.